Cargando…
Improving reliability estimation in cognitive diagnosis modeling
Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability e...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Springer US
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10615987/ https://www.ncbi.nlm.nih.gov/pubmed/36127563 http://dx.doi.org/10.3758/s13428-022-01967-5 |
_version_ | 1785129300493598720 |
---|---|
author | Kreitchmann, Rodrigo Schames de la Torre, Jimmy Sorrel, Miguel A. Nájera, Pablo Abad, Francisco J. |
author_facet | Kreitchmann, Rodrigo Schames de la Torre, Jimmy Sorrel, Miguel A. Nájera, Pablo Abad, Francisco J. |
author_sort | Kreitchmann, Rodrigo Schames |
collection | PubMed |
description | Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available |
format | Online Article Text |
id | pubmed-10615987 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Springer US |
record_format | MEDLINE/PubMed |
spelling | pubmed-106159872023-11-01 Improving reliability estimation in cognitive diagnosis modeling Kreitchmann, Rodrigo Schames de la Torre, Jimmy Sorrel, Miguel A. Nájera, Pablo Abad, Francisco J. Behav Res Methods Article Cognitive diagnosis models (CDMs) are used in educational, clinical, or personnel selection settings to classify respondents with respect to discrete attributes, identifying strengths and needs, and thus allowing to provide tailored training/treatment. As in any assessment, an accurate reliability estimation is crucial for valid score interpretations. In this sense, most CDM reliability indices are based on the posterior probabilities of the estimated attribute profiles. These posteriors are traditionally computed using point estimates for the model parameters as approximations to their populational values. If the uncertainty around these parameters is unaccounted for, the posteriors may be overly peaked, deriving into overestimated reliabilities. This article presents a multiple imputation (MI) procedure to integrate out the model parameters in the estimation of the posterior distributions, thus correcting the reliability estimation. A simulation study was conducted to compare the MI procedure with the traditional reliability estimation. Five factors were manipulated: the attribute structure, the CDM model (DINA and G-DINA), test length, sample size, and item quality. Additionally, an illustration using the Examination for the Certificate of Proficiency in English data was analyzed. The effect of sample size was studied by sampling subsets of subjects from the complete data. In both studies, the traditional reliability estimation systematically provided overestimated reliabilities, whereas the MI procedure offered more accurate results. Accordingly, practitioners in small educational or clinical settings should be aware that the reliability estimation using model parameter point estimates may be positively biased. R codes for the MI procedure are made available Springer US 2022-09-20 2023 /pmc/articles/PMC10615987/ /pubmed/36127563 http://dx.doi.org/10.3758/s13428-022-01967-5 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Article Kreitchmann, Rodrigo Schames de la Torre, Jimmy Sorrel, Miguel A. Nájera, Pablo Abad, Francisco J. Improving reliability estimation in cognitive diagnosis modeling |
title | Improving reliability estimation in cognitive diagnosis modeling |
title_full | Improving reliability estimation in cognitive diagnosis modeling |
title_fullStr | Improving reliability estimation in cognitive diagnosis modeling |
title_full_unstemmed | Improving reliability estimation in cognitive diagnosis modeling |
title_short | Improving reliability estimation in cognitive diagnosis modeling |
title_sort | improving reliability estimation in cognitive diagnosis modeling |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10615987/ https://www.ncbi.nlm.nih.gov/pubmed/36127563 http://dx.doi.org/10.3758/s13428-022-01967-5 |
work_keys_str_mv | AT kreitchmannrodrigoschames improvingreliabilityestimationincognitivediagnosismodeling AT delatorrejimmy improvingreliabilityestimationincognitivediagnosismodeling AT sorrelmiguela improvingreliabilityestimationincognitivediagnosismodeling AT najerapablo improvingreliabilityestimationincognitivediagnosismodeling AT abadfranciscoj improvingreliabilityestimationincognitivediagnosismodeling |