Cargando…
The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods
BACKGROUND: Item response theory (IRT) methods for addressing differential item functioning (DIF) can detect group differences in responses to individual items (e.g., bias). IRT and DIF-detection methods have been used increasingly often to identify bias in cognitive test performance by characterist...
Autores principales: | , , , , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8961895/ https://www.ncbi.nlm.nih.gov/pubmed/35346056 http://dx.doi.org/10.1186/s12874-022-01572-2 |
_version_ | 1784677686178742272 |
---|---|
author | Nichols, E. Deal, J. A. Swenor, B. K. Abraham, A. G. Armstrong, N. M. Bandeen-Roche, K. Carlson, M. C. Griswold, M. Lin, F. R. Mosley, T. H. Ramulu, P. Y. Reed, N. S. Sharrett, A. R. Gross, A. L. |
author_facet | Nichols, E. Deal, J. A. Swenor, B. K. Abraham, A. G. Armstrong, N. M. Bandeen-Roche, K. Carlson, M. C. Griswold, M. Lin, F. R. Mosley, T. H. Ramulu, P. Y. Reed, N. S. Sharrett, A. R. Gross, A. L. |
author_sort | Nichols, E. |
collection | PubMed |
description | BACKGROUND: Item response theory (IRT) methods for addressing differential item functioning (DIF) can detect group differences in responses to individual items (e.g., bias). IRT and DIF-detection methods have been used increasingly often to identify bias in cognitive test performance by characteristics (DIF grouping variables) such as hearing impairment, race, and educational attainment. Previous analyses have not considered the effect of missing data on inferences, although levels of missing cognitive data can be substantial in epidemiologic studies. METHODS: We used data from Visit 6 (2016–2017) of the Atherosclerosis Risk in Communities Neurocognitive Study (N = 3,580) to explicate the effect of artificially imposed missing data patterns and imputation on DIF detection. RESULTS: When missing data was imposed among individuals in a specific DIF group but was unrelated to cognitive test performance, there was no systematic error. However, when missing data was related to cognitive test performance and DIF group membership, there was systematic error in DIF detection. Given this missing data pattern, the median DIF detection error associated with 10%, 30%, and 50% missingness was -0.03, -0.08, and -0.14 standard deviation (SD) units without imputation, but this decreased to -0.02, -0.04, and -0.08 SD units with multiple imputation. CONCLUSIONS: Incorrect inferences in DIF testing have downstream consequences for the use of cognitive tests in research. It is therefore crucial to consider the effect and reasons behind missing data when evaluating bias in cognitive testing. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-022-01572-2. |
format | Online Article Text |
id | pubmed-8961895 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-89618952022-03-30 The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods Nichols, E. Deal, J. A. Swenor, B. K. Abraham, A. G. Armstrong, N. M. Bandeen-Roche, K. Carlson, M. C. Griswold, M. Lin, F. R. Mosley, T. H. Ramulu, P. Y. Reed, N. S. Sharrett, A. R. Gross, A. L. BMC Med Res Methodol Research BACKGROUND: Item response theory (IRT) methods for addressing differential item functioning (DIF) can detect group differences in responses to individual items (e.g., bias). IRT and DIF-detection methods have been used increasingly often to identify bias in cognitive test performance by characteristics (DIF grouping variables) such as hearing impairment, race, and educational attainment. Previous analyses have not considered the effect of missing data on inferences, although levels of missing cognitive data can be substantial in epidemiologic studies. METHODS: We used data from Visit 6 (2016–2017) of the Atherosclerosis Risk in Communities Neurocognitive Study (N = 3,580) to explicate the effect of artificially imposed missing data patterns and imputation on DIF detection. RESULTS: When missing data was imposed among individuals in a specific DIF group but was unrelated to cognitive test performance, there was no systematic error. However, when missing data was related to cognitive test performance and DIF group membership, there was systematic error in DIF detection. Given this missing data pattern, the median DIF detection error associated with 10%, 30%, and 50% missingness was -0.03, -0.08, and -0.14 standard deviation (SD) units without imputation, but this decreased to -0.02, -0.04, and -0.08 SD units with multiple imputation. CONCLUSIONS: Incorrect inferences in DIF testing have downstream consequences for the use of cognitive tests in research. It is therefore crucial to consider the effect and reasons behind missing data when evaluating bias in cognitive testing. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12874-022-01572-2. BioMed Central 2022-03-27 /pmc/articles/PMC8961895/ /pubmed/35346056 http://dx.doi.org/10.1186/s12874-022-01572-2 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
spellingShingle | Research Nichols, E. Deal, J. A. Swenor, B. K. Abraham, A. G. Armstrong, N. M. Bandeen-Roche, K. Carlson, M. C. Griswold, M. Lin, F. R. Mosley, T. H. Ramulu, P. Y. Reed, N. S. Sharrett, A. R. Gross, A. L. The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
title | The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
title_full | The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
title_fullStr | The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
title_full_unstemmed | The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
title_short | The effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
title_sort | effect of missing data and imputation on the detection of bias in cognitive testing using differential item functioning methods |
topic | Research |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8961895/ https://www.ncbi.nlm.nih.gov/pubmed/35346056 http://dx.doi.org/10.1186/s12874-022-01572-2 |
work_keys_str_mv | AT nicholse theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT dealja theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT swenorbk theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT abrahamag theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT armstrongnm theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT bandeenrochek theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT carlsonmc theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT griswoldm theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT linfr theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT mosleyth theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT ramulupy theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT reedns theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT sharrettar theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT grossal theeffectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT nicholse effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT dealja effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT swenorbk effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT abrahamag effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT armstrongnm effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT bandeenrochek effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT carlsonmc effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT griswoldm effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT linfr effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT mosleyth effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT ramulupy effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT reedns effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT sharrettar effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods AT grossal effectofmissingdataandimputationonthedetectionofbiasincognitivetestingusingdifferentialitemfunctioningmethods |