Cargando…

Implementing statistical equating for MRCP(UK) parts 1 and 2

BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the ch...

Descripción completa

Detalles Bibliográficos
Autores principales: McManus, IC, Chis, Liliana, Fox, Ray, Waller, Derek, Tang, Peter
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2014
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182791/
https://www.ncbi.nlm.nih.gov/pubmed/25257070
http://dx.doi.org/10.1186/1472-6920-14-204
_version_ 1782337608798437376
author McManus, IC
Chis, Liliana
Fox, Ray
Waller, Derek
Tang, Peter
author_facet McManus, IC
Chis, Liliana
Fox, Ray
Waller, Derek
Tang, Peter
author_sort McManus, IC
collection PubMed
description BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the change, the question of whether the pass rate increased amongst non-UK candidates, any possible role of Differential Item Functioning (DIF), and changes in examination predictive validity after the change. METHODS: Analysis of data of MRCP(UK) Part 1 exam from 2003 to 2013 and Part 2 exam from 2005 to 2013. RESULTS: Inspection suggested that Part 1 pass rates were stable after the introduction of statistical equating, but showed greater annual variation probably due to stronger candidates taking the examination earlier. Pass rates seemed to have increased in non-UK graduates after equating was introduced, but was not associated with any changes in DIF after statistical equating. Statistical modelling of the pass rates for non-UK graduates found that pass rates, in both Part 1 and Part 2, were increasing year on year, with the changes probably beginning before the introduction of equating. The predictive validity of Part 1 for Part 2 was higher with statistical equating than with the previous hybrid Angoff/Hofstee method, confirming the utility of IRT-based statistical equating. CONCLUSIONS: Statistical equating was successfully introduced into the MRCP(UK) Part 1 and Part 2 written examinations, resulting in higher predictive validity than the previous Angoff/Hofstee standard setting. Concerns about an artefactual increase in pass rates for non-UK candidates after equating were shown not to be well-founded. Most likely the changes resulted from a genuine increase in candidate ability, albeit for reasons which remain unclear, coupled with a cognitive illusion giving the impression of a step-change immediately after equating began. Statistical equating provides a robust standard-setting method, with a better theoretical foundation than judgemental techniques such as Angoff, and is more straightforward and requires far less examiner time to provide a more valid result. The present study provides a detailed case study of introducing statistical equating, and issues which may need to be considered with its introduction.
format Online
Article
Text
id pubmed-4182791
institution National Center for Biotechnology Information
language English
publishDate 2014
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-41827912014-10-03 Implementing statistical equating for MRCP(UK) parts 1 and 2 McManus, IC Chis, Liliana Fox, Ray Waller, Derek Tang, Peter BMC Med Educ Research Article BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the change, the question of whether the pass rate increased amongst non-UK candidates, any possible role of Differential Item Functioning (DIF), and changes in examination predictive validity after the change. METHODS: Analysis of data of MRCP(UK) Part 1 exam from 2003 to 2013 and Part 2 exam from 2005 to 2013. RESULTS: Inspection suggested that Part 1 pass rates were stable after the introduction of statistical equating, but showed greater annual variation probably due to stronger candidates taking the examination earlier. Pass rates seemed to have increased in non-UK graduates after equating was introduced, but was not associated with any changes in DIF after statistical equating. Statistical modelling of the pass rates for non-UK graduates found that pass rates, in both Part 1 and Part 2, were increasing year on year, with the changes probably beginning before the introduction of equating. The predictive validity of Part 1 for Part 2 was higher with statistical equating than with the previous hybrid Angoff/Hofstee method, confirming the utility of IRT-based statistical equating. CONCLUSIONS: Statistical equating was successfully introduced into the MRCP(UK) Part 1 and Part 2 written examinations, resulting in higher predictive validity than the previous Angoff/Hofstee standard setting. Concerns about an artefactual increase in pass rates for non-UK candidates after equating were shown not to be well-founded. Most likely the changes resulted from a genuine increase in candidate ability, albeit for reasons which remain unclear, coupled with a cognitive illusion giving the impression of a step-change immediately after equating began. Statistical equating provides a robust standard-setting method, with a better theoretical foundation than judgemental techniques such as Angoff, and is more straightforward and requires far less examiner time to provide a more valid result. The present study provides a detailed case study of introducing statistical equating, and issues which may need to be considered with its introduction. BioMed Central 2014-09-26 /pmc/articles/PMC4182791/ /pubmed/25257070 http://dx.doi.org/10.1186/1472-6920-14-204 Text en © McManus et al.; licensee BioMed Central Ltd. 2014 This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
spellingShingle Research Article
McManus, IC
Chis, Liliana
Fox, Ray
Waller, Derek
Tang, Peter
Implementing statistical equating for MRCP(UK) parts 1 and 2
title Implementing statistical equating for MRCP(UK) parts 1 and 2
title_full Implementing statistical equating for MRCP(UK) parts 1 and 2
title_fullStr Implementing statistical equating for MRCP(UK) parts 1 and 2
title_full_unstemmed Implementing statistical equating for MRCP(UK) parts 1 and 2
title_short Implementing statistical equating for MRCP(UK) parts 1 and 2
title_sort implementing statistical equating for mrcp(uk) parts 1 and 2
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182791/
https://www.ncbi.nlm.nih.gov/pubmed/25257070
http://dx.doi.org/10.1186/1472-6920-14-204
work_keys_str_mv AT mcmanusic implementingstatisticalequatingformrcpukparts1and2
AT chisliliana implementingstatisticalequatingformrcpukparts1and2
AT foxray implementingstatisticalequatingformrcpukparts1and2
AT wallerderek implementingstatisticalequatingformrcpukparts1and2
AT tangpeter implementingstatisticalequatingformrcpukparts1and2