Cargando…
Implementing statistical equating for MRCP(UK) parts 1 and 2
BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the ch...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2014
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182791/ https://www.ncbi.nlm.nih.gov/pubmed/25257070 http://dx.doi.org/10.1186/1472-6920-14-204 |
_version_ | 1782337608798437376 |
---|---|
author | McManus, IC Chis, Liliana Fox, Ray Waller, Derek Tang, Peter |
author_facet | McManus, IC Chis, Liliana Fox, Ray Waller, Derek Tang, Peter |
author_sort | McManus, IC |
collection | PubMed |
description | BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the change, the question of whether the pass rate increased amongst non-UK candidates, any possible role of Differential Item Functioning (DIF), and changes in examination predictive validity after the change. METHODS: Analysis of data of MRCP(UK) Part 1 exam from 2003 to 2013 and Part 2 exam from 2005 to 2013. RESULTS: Inspection suggested that Part 1 pass rates were stable after the introduction of statistical equating, but showed greater annual variation probably due to stronger candidates taking the examination earlier. Pass rates seemed to have increased in non-UK graduates after equating was introduced, but was not associated with any changes in DIF after statistical equating. Statistical modelling of the pass rates for non-UK graduates found that pass rates, in both Part 1 and Part 2, were increasing year on year, with the changes probably beginning before the introduction of equating. The predictive validity of Part 1 for Part 2 was higher with statistical equating than with the previous hybrid Angoff/Hofstee method, confirming the utility of IRT-based statistical equating. CONCLUSIONS: Statistical equating was successfully introduced into the MRCP(UK) Part 1 and Part 2 written examinations, resulting in higher predictive validity than the previous Angoff/Hofstee standard setting. Concerns about an artefactual increase in pass rates for non-UK candidates after equating were shown not to be well-founded. Most likely the changes resulted from a genuine increase in candidate ability, albeit for reasons which remain unclear, coupled with a cognitive illusion giving the impression of a step-change immediately after equating began. Statistical equating provides a robust standard-setting method, with a better theoretical foundation than judgemental techniques such as Angoff, and is more straightforward and requires far less examiner time to provide a more valid result. The present study provides a detailed case study of introducing statistical equating, and issues which may need to be considered with its introduction. |
format | Online Article Text |
id | pubmed-4182791 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2014 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-41827912014-10-03 Implementing statistical equating for MRCP(UK) parts 1 and 2 McManus, IC Chis, Liliana Fox, Ray Waller, Derek Tang, Peter BMC Med Educ Research Article BACKGROUND: The MRCP(UK) exam, in 2008 and 2010, changed the standard-setting of its Part 1 and Part 2 examinations from a hybrid Angoff/Hofstee method to statistical equating using Item Response Theory, the reference group being UK graduates. The present paper considers the implementation of the change, the question of whether the pass rate increased amongst non-UK candidates, any possible role of Differential Item Functioning (DIF), and changes in examination predictive validity after the change. METHODS: Analysis of data of MRCP(UK) Part 1 exam from 2003 to 2013 and Part 2 exam from 2005 to 2013. RESULTS: Inspection suggested that Part 1 pass rates were stable after the introduction of statistical equating, but showed greater annual variation probably due to stronger candidates taking the examination earlier. Pass rates seemed to have increased in non-UK graduates after equating was introduced, but was not associated with any changes in DIF after statistical equating. Statistical modelling of the pass rates for non-UK graduates found that pass rates, in both Part 1 and Part 2, were increasing year on year, with the changes probably beginning before the introduction of equating. The predictive validity of Part 1 for Part 2 was higher with statistical equating than with the previous hybrid Angoff/Hofstee method, confirming the utility of IRT-based statistical equating. CONCLUSIONS: Statistical equating was successfully introduced into the MRCP(UK) Part 1 and Part 2 written examinations, resulting in higher predictive validity than the previous Angoff/Hofstee standard setting. Concerns about an artefactual increase in pass rates for non-UK candidates after equating were shown not to be well-founded. Most likely the changes resulted from a genuine increase in candidate ability, albeit for reasons which remain unclear, coupled with a cognitive illusion giving the impression of a step-change immediately after equating began. Statistical equating provides a robust standard-setting method, with a better theoretical foundation than judgemental techniques such as Angoff, and is more straightforward and requires far less examiner time to provide a more valid result. The present study provides a detailed case study of introducing statistical equating, and issues which may need to be considered with its introduction. BioMed Central 2014-09-26 /pmc/articles/PMC4182791/ /pubmed/25257070 http://dx.doi.org/10.1186/1472-6920-14-204 Text en © McManus et al.; licensee BioMed Central Ltd. 2014 This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. |
spellingShingle | Research Article McManus, IC Chis, Liliana Fox, Ray Waller, Derek Tang, Peter Implementing statistical equating for MRCP(UK) parts 1 and 2 |
title | Implementing statistical equating for MRCP(UK) parts 1 and 2 |
title_full | Implementing statistical equating for MRCP(UK) parts 1 and 2 |
title_fullStr | Implementing statistical equating for MRCP(UK) parts 1 and 2 |
title_full_unstemmed | Implementing statistical equating for MRCP(UK) parts 1 and 2 |
title_short | Implementing statistical equating for MRCP(UK) parts 1 and 2 |
title_sort | implementing statistical equating for mrcp(uk) parts 1 and 2 |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4182791/ https://www.ncbi.nlm.nih.gov/pubmed/25257070 http://dx.doi.org/10.1186/1472-6920-14-204 |
work_keys_str_mv | AT mcmanusic implementingstatisticalequatingformrcpukparts1and2 AT chisliliana implementingstatisticalequatingformrcpukparts1and2 AT foxray implementingstatisticalequatingformrcpukparts1and2 AT wallerderek implementingstatisticalequatingformrcpukparts1and2 AT tangpeter implementingstatisticalequatingformrcpukparts1and2 |