Cargando…

Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods

BACKGROUND: Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees’ responses have to be analyzed and scored to derive infor...

Descripción completa

Detalles Bibliográficos
Autores principales: Kanzow, Amelie Friederike, Schmidt, Dennis, Kanzow, Philipp
Formato: Online Artículo Texto
Lenguaje:English
Publicado: JMIR Publications 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10238964/
https://www.ncbi.nlm.nih.gov/pubmed/37001510
http://dx.doi.org/10.2196/44084
_version_ 1785053395710640128
author Kanzow, Amelie Friederike
Schmidt, Dennis
Kanzow, Philipp
author_facet Kanzow, Amelie Friederike
Schmidt, Dennis
Kanzow, Philipp
author_sort Kanzow, Amelie Friederike
collection PubMed
description BACKGROUND: Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees’ responses have to be analyzed and scored to derive information about examinees’ true knowledge. OBJECTIVE: The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees’ true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived. METHODS: Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees’ true knowledge using fictitious examinations with 100 single-choice items were calculated. RESULTS: A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between –3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between –1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between –2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees’ true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between –60.0% (95% CI –60% to –60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5. CONCLUSIONS: In examinations with single-choice items, the scoring result is not always equivalent to examinees’ true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used.
format Online
Article
Text
id pubmed-10238964
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher JMIR Publications
record_format MEDLINE/PubMed
spelling pubmed-102389642023-06-04 Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods Kanzow, Amelie Friederike Schmidt, Dennis Kanzow, Philipp JMIR Med Educ Review BACKGROUND: Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees’ responses have to be analyzed and scored to derive information about examinees’ true knowledge. OBJECTIVE: The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees’ true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived. METHODS: Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees’ true knowledge using fictitious examinations with 100 single-choice items were calculated. RESULTS: A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between –3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between –1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between –2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees’ true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between –60.0% (95% CI –60% to –60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5. CONCLUSIONS: In examinations with single-choice items, the scoring result is not always equivalent to examinees’ true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used. JMIR Publications 2023-05-19 /pmc/articles/PMC10238964/ /pubmed/37001510 http://dx.doi.org/10.2196/44084 Text en ©Amelie Friederike Kanzow, Dennis Schmidt, Philipp Kanzow. Originally published in JMIR Medical Education (https://mededu.jmir.org), 19.05.2023. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/, as well as this copyright and license information must be included.
spellingShingle Review
Kanzow, Amelie Friederike
Schmidt, Dennis
Kanzow, Philipp
Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
title Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
title_full Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
title_fullStr Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
title_full_unstemmed Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
title_short Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods
title_sort scoring single-response multiple-choice items: scoping review and comparison of different scoring methods
topic Review
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10238964/
https://www.ncbi.nlm.nih.gov/pubmed/37001510
http://dx.doi.org/10.2196/44084
work_keys_str_mv AT kanzowameliefriederike scoringsingleresponsemultiplechoiceitemsscopingreviewandcomparisonofdifferentscoringmethods
AT schmidtdennis scoringsingleresponsemultiplechoiceitemsscopingreviewandcomparisonofdifferentscoringmethods
AT kanzowphilipp scoringsingleresponsemultiplechoiceitemsscopingreviewandcomparisonofdifferentscoringmethods