Cargando…

The reliability of the pass/fail decision for assessments comprised of multiple components

Objective: The decision having the most serious consequences for a student taking an assessment is the one to pass or fail that student. For this reason, the reliability of the pass/fail decision must be determined for high quality assessments, just as the measurement reliability of the point values...

Descripción completa

Detalles Bibliográficos
Autores principales: Möltner, Andreas, Tımbıl, Sevgi, Jünger, Jana
Formato: Online Artículo Texto
Lenguaje:English
Publicado: German Medical Science GMS Publishing House 2015
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4606479/
https://www.ncbi.nlm.nih.gov/pubmed/26483855
http://dx.doi.org/10.3205/zma000984
_version_ 1782395358585815040
author Möltner, Andreas
Tımbıl, Sevgi
Jünger, Jana
author_facet Möltner, Andreas
Tımbıl, Sevgi
Jünger, Jana
author_sort Möltner, Andreas
collection PubMed
description Objective: The decision having the most serious consequences for a student taking an assessment is the one to pass or fail that student. For this reason, the reliability of the pass/fail decision must be determined for high quality assessments, just as the measurement reliability of the point values. Assessments in a particular subject (graded course credit) are often composed of multiple components that must be passed independently of each other. When “conjunctively” combining separate pass/fail decisions, as with other complex decision rules for passing, adequate methods of analysis are necessary for estimating the accuracy and consistency of these classifications. To date, very few papers have addressed this issue; a generally applicable procedure was published by Douglas and Mislevy in 2010. Using the example of an assessment comprised of several parts that must be passed separately, this study analyzes the reliability underlying the decision to pass or fail students and discusses the impact of an improved method for identifying those who do not fulfill the minimum requirements. Method: The accuracy and consistency of the decision to pass or fail an examinee in the subject cluster Internal Medicine/General Medicine/Clinical Chemistry at the University of Heidelberg’s Faculty of Medicine was investigated. This cluster requires students to separately pass three components (two written exams and an OSCE), whereby students may reattempt to pass each component twice. Our analysis was carried out using the method described by Douglas and Mislevy. Results: Frequently, when complex logical connections exist between the individual pass/fail decisions in the case of low failure rates, only a very low reliability for the overall decision to grant graded course credit can be achieved, even if high reliabilities exist for the various components. For the example analyzed here, the classification accuracy and consistency when conjunctively combining the three individual parts is relatively low with κ=0.49 or κ=0.47, despite the good reliability of over 0.75 for each of the three components. The option to repeat each component twice leads to a situation in which only about half of the candidates who do not satisfy the minimum requirements would fail the overall assessment, while the other half is able to continue their studies despite having deficient knowledge and skills. Conclusion: The method put forth by Douglas and Mislevy allows the analysis of the decision accuracy and consistency for complex combinations of scores from different components. Even in the case of highly reliable components, it is not necessarily so that a reliable pass/fail decision has been reached – for instance in the case of low failure rates. Assessments must be administered with the explicit goal of identifying examinees that do not fulfill the minimum requirements.
format Online
Article
Text
id pubmed-4606479
institution National Center for Biotechnology Information
language English
publishDate 2015
publisher German Medical Science GMS Publishing House
record_format MEDLINE/PubMed
spelling pubmed-46064792015-10-19 The reliability of the pass/fail decision for assessments comprised of multiple components Möltner, Andreas Tımbıl, Sevgi Jünger, Jana GMS Z Med Ausbild Article Objective: The decision having the most serious consequences for a student taking an assessment is the one to pass or fail that student. For this reason, the reliability of the pass/fail decision must be determined for high quality assessments, just as the measurement reliability of the point values. Assessments in a particular subject (graded course credit) are often composed of multiple components that must be passed independently of each other. When “conjunctively” combining separate pass/fail decisions, as with other complex decision rules for passing, adequate methods of analysis are necessary for estimating the accuracy and consistency of these classifications. To date, very few papers have addressed this issue; a generally applicable procedure was published by Douglas and Mislevy in 2010. Using the example of an assessment comprised of several parts that must be passed separately, this study analyzes the reliability underlying the decision to pass or fail students and discusses the impact of an improved method for identifying those who do not fulfill the minimum requirements. Method: The accuracy and consistency of the decision to pass or fail an examinee in the subject cluster Internal Medicine/General Medicine/Clinical Chemistry at the University of Heidelberg’s Faculty of Medicine was investigated. This cluster requires students to separately pass three components (two written exams and an OSCE), whereby students may reattempt to pass each component twice. Our analysis was carried out using the method described by Douglas and Mislevy. Results: Frequently, when complex logical connections exist between the individual pass/fail decisions in the case of low failure rates, only a very low reliability for the overall decision to grant graded course credit can be achieved, even if high reliabilities exist for the various components. For the example analyzed here, the classification accuracy and consistency when conjunctively combining the three individual parts is relatively low with κ=0.49 or κ=0.47, despite the good reliability of over 0.75 for each of the three components. The option to repeat each component twice leads to a situation in which only about half of the candidates who do not satisfy the minimum requirements would fail the overall assessment, while the other half is able to continue their studies despite having deficient knowledge and skills. Conclusion: The method put forth by Douglas and Mislevy allows the analysis of the decision accuracy and consistency for complex combinations of scores from different components. Even in the case of highly reliable components, it is not necessarily so that a reliable pass/fail decision has been reached – for instance in the case of low failure rates. Assessments must be administered with the explicit goal of identifying examinees that do not fulfill the minimum requirements. German Medical Science GMS Publishing House 2015-10-15 /pmc/articles/PMC4606479/ /pubmed/26483855 http://dx.doi.org/10.3205/zma000984 Text en Copyright © 2015 Möltner et al. http://creativecommons.org/licenses/by/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 License.
spellingShingle Article
Möltner, Andreas
Tımbıl, Sevgi
Jünger, Jana
The reliability of the pass/fail decision for assessments comprised of multiple components
title The reliability of the pass/fail decision for assessments comprised of multiple components
title_full The reliability of the pass/fail decision for assessments comprised of multiple components
title_fullStr The reliability of the pass/fail decision for assessments comprised of multiple components
title_full_unstemmed The reliability of the pass/fail decision for assessments comprised of multiple components
title_short The reliability of the pass/fail decision for assessments comprised of multiple components
title_sort reliability of the pass/fail decision for assessments comprised of multiple components
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4606479/
https://www.ncbi.nlm.nih.gov/pubmed/26483855
http://dx.doi.org/10.3205/zma000984
work_keys_str_mv AT moltnerandreas thereliabilityofthepassfaildecisionforassessmentscomprisedofmultiplecomponents
AT tımbılsevgi thereliabilityofthepassfaildecisionforassessmentscomprisedofmultiplecomponents
AT jungerjana thereliabilityofthepassfaildecisionforassessmentscomprisedofmultiplecomponents
AT moltnerandreas reliabilityofthepassfaildecisionforassessmentscomprisedofmultiplecomponents
AT tımbılsevgi reliabilityofthepassfaildecisionforassessmentscomprisedofmultiplecomponents
AT jungerjana reliabilityofthepassfaildecisionforassessmentscomprisedofmultiplecomponents