Cargando…

Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis

BACKGROUND: Progress testing is an assessment tool used to periodically assess all students at the end-of-curriculum level. Because students cannot know everything, it is important that they recognize their lack of knowledge. For that reason, the formula-scoring method has usually been used. However...

Descripción completa

Detalles Bibliográficos
Autores principales: Cecilio-Fernandes, Dario, Medema, Harro, Collares, Carlos Fernando, Schuwirth, Lambert, Cohen-Schotanus, Janke, Tio, René A.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2017
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5679154/
https://www.ncbi.nlm.nih.gov/pubmed/29121888
http://dx.doi.org/10.1186/s12909-017-1051-8
_version_ 1783277554404163584
author Cecilio-Fernandes, Dario
Medema, Harro
Collares, Carlos Fernando
Schuwirth, Lambert
Cohen-Schotanus, Janke
Tio, René A.
author_facet Cecilio-Fernandes, Dario
Medema, Harro
Collares, Carlos Fernando
Schuwirth, Lambert
Cohen-Schotanus, Janke
Tio, René A.
author_sort Cecilio-Fernandes, Dario
collection PubMed
description BACKGROUND: Progress testing is an assessment tool used to periodically assess all students at the end-of-curriculum level. Because students cannot know everything, it is important that they recognize their lack of knowledge. For that reason, the formula-scoring method has usually been used. However, where partial knowledge needs to be taken into account, the number-right scoring method is used. Research comparing both methods has yielded conflicting results. As far as we know, in all these studies, Classical Test Theory or Generalizability Theory was used to analyze the data. In contrast to these studies, we will explore the use of the Rasch model to compare both methods. METHODS: A 2 × 2 crossover design was used in a study where 298 students from four medical schools participated. A sample of 200 previously used questions from the progress tests was selected. The data were analyzed using the Rasch model, which provides fit parameters, reliability coefficients, and response option analysis. RESULTS: The fit parameters were in the optimal interval ranging from 0.50 to 1.50, and the means were around 1.00. The person and item reliability coefficients were higher in the number-right condition than in the formula-scoring condition. The response option analysis showed that the majority of dysfunctional items emerged in the formula-scoring condition. CONCLUSIONS: The findings of this study support the use of number-right scoring over formula scoring. Rasch model analyses showed that tests with number-right scoring have better psychometric properties than formula scoring. However, choosing the appropriate scoring method should depend not only on psychometric properties but also on self-directed test-taking strategies and metacognitive skills.
format Online
Article
Text
id pubmed-5679154
institution National Center for Biotechnology Information
language English
publishDate 2017
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-56791542017-11-17 Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis Cecilio-Fernandes, Dario Medema, Harro Collares, Carlos Fernando Schuwirth, Lambert Cohen-Schotanus, Janke Tio, René A. BMC Med Educ Research Article BACKGROUND: Progress testing is an assessment tool used to periodically assess all students at the end-of-curriculum level. Because students cannot know everything, it is important that they recognize their lack of knowledge. For that reason, the formula-scoring method has usually been used. However, where partial knowledge needs to be taken into account, the number-right scoring method is used. Research comparing both methods has yielded conflicting results. As far as we know, in all these studies, Classical Test Theory or Generalizability Theory was used to analyze the data. In contrast to these studies, we will explore the use of the Rasch model to compare both methods. METHODS: A 2 × 2 crossover design was used in a study where 298 students from four medical schools participated. A sample of 200 previously used questions from the progress tests was selected. The data were analyzed using the Rasch model, which provides fit parameters, reliability coefficients, and response option analysis. RESULTS: The fit parameters were in the optimal interval ranging from 0.50 to 1.50, and the means were around 1.00. The person and item reliability coefficients were higher in the number-right condition than in the formula-scoring condition. The response option analysis showed that the majority of dysfunctional items emerged in the formula-scoring condition. CONCLUSIONS: The findings of this study support the use of number-right scoring over formula scoring. Rasch model analyses showed that tests with number-right scoring have better psychometric properties than formula scoring. However, choosing the appropriate scoring method should depend not only on psychometric properties but also on self-directed test-taking strategies and metacognitive skills. BioMed Central 2017-11-09 /pmc/articles/PMC5679154/ /pubmed/29121888 http://dx.doi.org/10.1186/s12909-017-1051-8 Text en © The Author(s). 2017 Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
spellingShingle Research Article
Cecilio-Fernandes, Dario
Medema, Harro
Collares, Carlos Fernando
Schuwirth, Lambert
Cohen-Schotanus, Janke
Tio, René A.
Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
title Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
title_full Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
title_fullStr Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
title_full_unstemmed Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
title_short Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis
title_sort comparison of formula and number-right scoring in undergraduate medical training: a rasch model analysis
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5679154/
https://www.ncbi.nlm.nih.gov/pubmed/29121888
http://dx.doi.org/10.1186/s12909-017-1051-8
work_keys_str_mv AT ceciliofernandesdario comparisonofformulaandnumberrightscoringinundergraduatemedicaltrainingaraschmodelanalysis
AT medemaharro comparisonofformulaandnumberrightscoringinundergraduatemedicaltrainingaraschmodelanalysis
AT collarescarlosfernando comparisonofformulaandnumberrightscoringinundergraduatemedicaltrainingaraschmodelanalysis
AT schuwirthlambert comparisonofformulaandnumberrightscoringinundergraduatemedicaltrainingaraschmodelanalysis
AT cohenschotanusjanke comparisonofformulaandnumberrightscoringinundergraduatemedicaltrainingaraschmodelanalysis
AT tiorenea comparisonofformulaandnumberrightscoringinundergraduatemedicaltrainingaraschmodelanalysis