Cargando…
Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach
INTRODUCTION: The Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from vari...
Autores principales: | , , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Department of Emergency Medicine, University of California, Irvine School of Medicine
2019
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6324708/ https://www.ncbi.nlm.nih.gov/pubmed/30643603 http://dx.doi.org/10.5811/westjem.2018.12.39844 |
_version_ | 1783386021465948160 |
---|---|
author | Jong, Michael Elliott, Nicole Nguyen, Michael Goyke, Terrence Johnson, Steven Cook, Matthew Lindauer, Lisa Best, Katie Gernerd, Douglas Morolla, Louis Matuzsan, Zachary Kane, Bryan |
author_facet | Jong, Michael Elliott, Nicole Nguyen, Michael Goyke, Terrence Johnson, Steven Cook, Matthew Lindauer, Lisa Best, Katie Gernerd, Douglas Morolla, Louis Matuzsan, Zachary Kane, Bryan |
author_sort | Jong, Michael |
collection | PubMed |
description | INTRODUCTION: The Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen’s Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case. METHODS: Prospectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1–4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1–5 Likert anchors used by the QSAT. A PGY 2–4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions. RESULTS: We enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572–0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733–0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation. CONCLUSION: In this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees. |
format | Online Article Text |
id | pubmed-6324708 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2019 |
publisher | Department of Emergency Medicine, University of California, Irvine School of Medicine |
record_format | MEDLINE/PubMed |
spelling | pubmed-63247082019-01-14 Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach Jong, Michael Elliott, Nicole Nguyen, Michael Goyke, Terrence Johnson, Steven Cook, Matthew Lindauer, Lisa Best, Katie Gernerd, Douglas Morolla, Louis Matuzsan, Zachary Kane, Bryan West J Emerg Med Original Research INTRODUCTION: The Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen’s Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case. METHODS: Prospectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1–4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1–5 Likert anchors used by the QSAT. A PGY 2–4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions. RESULTS: We enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572–0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733–0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation. CONCLUSION: In this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees. Department of Emergency Medicine, University of California, Irvine School of Medicine 2019-01 2018-12-17 /pmc/articles/PMC6324708/ /pubmed/30643603 http://dx.doi.org/10.5811/westjem.2018.12.39844 Text en Copyright: © 2019 Jong et al. http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) License. See: http://creativecommons.org/licenses/by/4.0/ |
spellingShingle | Original Research Jong, Michael Elliott, Nicole Nguyen, Michael Goyke, Terrence Johnson, Steven Cook, Matthew Lindauer, Lisa Best, Katie Gernerd, Douglas Morolla, Louis Matuzsan, Zachary Kane, Bryan Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach |
title | Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach |
title_full | Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach |
title_fullStr | Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach |
title_full_unstemmed | Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach |
title_short | Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach |
title_sort | assessment of emergency medicine resident performance in an adult simulation using a multisource feedback approach |
topic | Original Research |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6324708/ https://www.ncbi.nlm.nih.gov/pubmed/30643603 http://dx.doi.org/10.5811/westjem.2018.12.39844 |
work_keys_str_mv | AT jongmichael assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT elliottnicole assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT nguyenmichael assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT goyketerrence assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT johnsonsteven assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT cookmatthew assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT lindauerlisa assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT bestkatie assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT gernerddouglas assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT morollalouis assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT matuzsanzachary assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach AT kanebryan assessmentofemergencymedicineresidentperformanceinanadultsimulationusingamultisourcefeedbackapproach |