Cargando…

Insights into the Angoff method: results from a simulation study

BACKGROUND: In standard setting techniques involving panels of judges, the attributes of judges may affect the cut-scores. This simulation study modelled the effect of the number of judges and test items, as well as the impact of judges’ attributes such as accuracy, stringency and influence on other...

Descripción completa

Detalles Bibliográficos
Autores principales: Shulruf, Boaz, Wilkinson, Tim, Weller, Jennifer, Jones, Philip, Poole, Phillippa
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2016
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4855704/
https://www.ncbi.nlm.nih.gov/pubmed/27142788
http://dx.doi.org/10.1186/s12909-016-0656-7
_version_ 1782430397503635456
author Shulruf, Boaz
Wilkinson, Tim
Weller, Jennifer
Jones, Philip
Poole, Phillippa
author_facet Shulruf, Boaz
Wilkinson, Tim
Weller, Jennifer
Jones, Philip
Poole, Phillippa
author_sort Shulruf, Boaz
collection PubMed
description BACKGROUND: In standard setting techniques involving panels of judges, the attributes of judges may affect the cut-scores. This simulation study modelled the effect of the number of judges and test items, as well as the impact of judges’ attributes such as accuracy, stringency and influence on others on the precision of the cut-scores. METHODS: Forty nine combinations of Angoff panels (N = 5, 10, 15, 20, 30, 50, and 80) and test items (n = 5, 10, 15, 20, 30, 50, and 80) were simulated. Each combination was simulated 100 times (in total 4,900 simulations). The simulation was of judges attributes: stringency, accuracy and leadership. Impact of judges attributes, number of judges, number of test items and Angoff’s second (compared to the first) round on the precision of a panel’s cut-score was measured by the deviation of the panel’s cut-score from the cut-score’s true value. RESULTS: Findings from 4900 simulated panels supported Angoff being both reliable and valid. Unless the number of test items is small, panels of around 15 judges with mixed levels of expertise provide the most precise estimates. Furthermore, if test data were not presented, a second round of decision-making, as used in the modified Angoff, adds little to precision. A panel which has only experts or only non-experts yields a cut-score which is less precise than a cut-score yielded by a mixed-expertise panel, suggesting that optimal composition of an Angoff panel should include a range of judges with diverse expertise and stringency. CONCLUSIONS: Simulations aim to improve our understanding of the models assessed but they do not describe natural phenomena as they do not use observed data. While the simulations undertaken in this study help clarify how to set cut-scores defensibly, it is essential to confirm these theories in practice.
format Online
Article
Text
id pubmed-4855704
institution National Center for Biotechnology Information
language English
publishDate 2016
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-48557042016-05-05 Insights into the Angoff method: results from a simulation study Shulruf, Boaz Wilkinson, Tim Weller, Jennifer Jones, Philip Poole, Phillippa BMC Med Educ Research Article BACKGROUND: In standard setting techniques involving panels of judges, the attributes of judges may affect the cut-scores. This simulation study modelled the effect of the number of judges and test items, as well as the impact of judges’ attributes such as accuracy, stringency and influence on others on the precision of the cut-scores. METHODS: Forty nine combinations of Angoff panels (N = 5, 10, 15, 20, 30, 50, and 80) and test items (n = 5, 10, 15, 20, 30, 50, and 80) were simulated. Each combination was simulated 100 times (in total 4,900 simulations). The simulation was of judges attributes: stringency, accuracy and leadership. Impact of judges attributes, number of judges, number of test items and Angoff’s second (compared to the first) round on the precision of a panel’s cut-score was measured by the deviation of the panel’s cut-score from the cut-score’s true value. RESULTS: Findings from 4900 simulated panels supported Angoff being both reliable and valid. Unless the number of test items is small, panels of around 15 judges with mixed levels of expertise provide the most precise estimates. Furthermore, if test data were not presented, a second round of decision-making, as used in the modified Angoff, adds little to precision. A panel which has only experts or only non-experts yields a cut-score which is less precise than a cut-score yielded by a mixed-expertise panel, suggesting that optimal composition of an Angoff panel should include a range of judges with diverse expertise and stringency. CONCLUSIONS: Simulations aim to improve our understanding of the models assessed but they do not describe natural phenomena as they do not use observed data. While the simulations undertaken in this study help clarify how to set cut-scores defensibly, it is essential to confirm these theories in practice. BioMed Central 2016-05-04 /pmc/articles/PMC4855704/ /pubmed/27142788 http://dx.doi.org/10.1186/s12909-016-0656-7 Text en © Shulruf et al. 2016 Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
spellingShingle Research Article
Shulruf, Boaz
Wilkinson, Tim
Weller, Jennifer
Jones, Philip
Poole, Phillippa
Insights into the Angoff method: results from a simulation study
title Insights into the Angoff method: results from a simulation study
title_full Insights into the Angoff method: results from a simulation study
title_fullStr Insights into the Angoff method: results from a simulation study
title_full_unstemmed Insights into the Angoff method: results from a simulation study
title_short Insights into the Angoff method: results from a simulation study
title_sort insights into the angoff method: results from a simulation study
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4855704/
https://www.ncbi.nlm.nih.gov/pubmed/27142788
http://dx.doi.org/10.1186/s12909-016-0656-7
work_keys_str_mv AT shulrufboaz insightsintotheangoffmethodresultsfromasimulationstudy
AT wilkinsontim insightsintotheangoffmethodresultsfromasimulationstudy
AT wellerjennifer insightsintotheangoffmethodresultsfromasimulationstudy
AT jonesphilip insightsintotheangoffmethodresultsfromasimulationstudy
AT poolephillippa insightsintotheangoffmethodresultsfromasimulationstudy