Cargando…
Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination
BACKGROUND: The American Board of Emergency Medicine (ABEM) in‐person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID‐19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020. OBJECTIVES: The purpose of this investi...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
John Wiley and Sons Inc.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10041069/ https://www.ncbi.nlm.nih.gov/pubmed/36994316 http://dx.doi.org/10.1002/aet2.10850 |
_version_ | 1784912624952016896 |
---|---|
author | Chudnofsky, Carl R. Reisdorff, Earl J. Joldersma, Kevin B. Ruff, Kathleen C. Goyal, Deepi G. Gorgas, Diane L. |
author_facet | Chudnofsky, Carl R. Reisdorff, Earl J. Joldersma, Kevin B. Ruff, Kathleen C. Goyal, Deepi G. Gorgas, Diane L. |
author_sort | Chudnofsky, Carl R. |
collection | PubMed |
description | BACKGROUND: The American Board of Emergency Medicine (ABEM) in‐person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID‐19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020. OBJECTIVES: The purpose of this investigation was to determine whether there was sufficient validity and reliability evidence to support the continued use of the ABEM virtual Oral Examination (VOE) for certification decisions. METHODS: This retrospective, descriptive study used multiple data sources to provide validity evidence and reliability data. Validity evidence focused on test content, response processes, internal structure (e.g., internal consistency and item response theory), and the consequences of testing. A multifaceted Rasch reliability coefficient was used to measure reliability. Study data were from two 2019 in‐person OCEs and the first four VOE administrations. RESULTS: There were 2279 physicians who took the 2019 in‐person OCE examination and 2153 physicians who took the VOE during the study period. Among the OCE group, 92.0% agreed or strongly agreed that the cases on the examination were cases that an emergency physician should be expected to see; 91.1% of the VOE group agreed or strongly agreed. A similar pattern of responses given to a question about whether the cases on the examination were cases that they had seen. Additional evidence of validity was obtained by the use of the EM Model, the process for case development, the use of think‐aloud protocols, and similar test performance patterns (e.g., pass rates). For reliability, the Rasch reliability coefficients for the OCE and the VOE during the study period were all >0.90. CONCLUSIONS: There was substantial validity evidence and reliability to support ongoing use of the ABEM VOE to make confident and defensible certification decisions. |
format | Online Article Text |
id | pubmed-10041069 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | John Wiley and Sons Inc. |
record_format | MEDLINE/PubMed |
spelling | pubmed-100410692023-04-01 Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination Chudnofsky, Carl R. Reisdorff, Earl J. Joldersma, Kevin B. Ruff, Kathleen C. Goyal, Deepi G. Gorgas, Diane L. AEM Educ Train Original Contribution BACKGROUND: The American Board of Emergency Medicine (ABEM) in‐person Oral Certification Examination (OCE) was halted abruptly in 2020 due to the COVID‐19 pandemic. The OCE was reconfigured to be administered in a virtual environment starting in December 2020. OBJECTIVES: The purpose of this investigation was to determine whether there was sufficient validity and reliability evidence to support the continued use of the ABEM virtual Oral Examination (VOE) for certification decisions. METHODS: This retrospective, descriptive study used multiple data sources to provide validity evidence and reliability data. Validity evidence focused on test content, response processes, internal structure (e.g., internal consistency and item response theory), and the consequences of testing. A multifaceted Rasch reliability coefficient was used to measure reliability. Study data were from two 2019 in‐person OCEs and the first four VOE administrations. RESULTS: There were 2279 physicians who took the 2019 in‐person OCE examination and 2153 physicians who took the VOE during the study period. Among the OCE group, 92.0% agreed or strongly agreed that the cases on the examination were cases that an emergency physician should be expected to see; 91.1% of the VOE group agreed or strongly agreed. A similar pattern of responses given to a question about whether the cases on the examination were cases that they had seen. Additional evidence of validity was obtained by the use of the EM Model, the process for case development, the use of think‐aloud protocols, and similar test performance patterns (e.g., pass rates). For reliability, the Rasch reliability coefficients for the OCE and the VOE during the study period were all >0.90. CONCLUSIONS: There was substantial validity evidence and reliability to support ongoing use of the ABEM VOE to make confident and defensible certification decisions. John Wiley and Sons Inc. 2023-03-26 /pmc/articles/PMC10041069/ /pubmed/36994316 http://dx.doi.org/10.1002/aet2.10850 Text en © 2023 The Authors. AEM Education and Training published by Wiley Periodicals LLC on behalf of Society for Academic Emergency Medicine. https://creativecommons.org/licenses/by/4.0/This is an open access article under the terms of the http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. |
spellingShingle | Original Contribution Chudnofsky, Carl R. Reisdorff, Earl J. Joldersma, Kevin B. Ruff, Kathleen C. Goyal, Deepi G. Gorgas, Diane L. Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination |
title | Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination |
title_full | Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination |
title_fullStr | Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination |
title_full_unstemmed | Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination |
title_short | Early validity and reliability evidence for the American Board of Emergency Medicine Virtual Oral Examination |
title_sort | early validity and reliability evidence for the american board of emergency medicine virtual oral examination |
topic | Original Contribution |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10041069/ https://www.ncbi.nlm.nih.gov/pubmed/36994316 http://dx.doi.org/10.1002/aet2.10850 |
work_keys_str_mv | AT chudnofskycarlr earlyvalidityandreliabilityevidencefortheamericanboardofemergencymedicinevirtualoralexamination AT reisdorffearlj earlyvalidityandreliabilityevidencefortheamericanboardofemergencymedicinevirtualoralexamination AT joldersmakevinb earlyvalidityandreliabilityevidencefortheamericanboardofemergencymedicinevirtualoralexamination AT ruffkathleenc earlyvalidityandreliabilityevidencefortheamericanboardofemergencymedicinevirtualoralexamination AT goyaldeepig earlyvalidityandreliabilityevidencefortheamericanboardofemergencymedicinevirtualoralexamination AT gorgasdianel earlyvalidityandreliabilityevidencefortheamericanboardofemergencymedicinevirtualoralexamination |