Cargando…

Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records

BACKGROUND: Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depress...

Descripción completa

Detalles Bibliográficos
Autores principales: Nickson, David, Singmann, Henrik, Meyer, Caroline, Toro, Carla, Walasek, Lukasz
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10696659/
https://www.ncbi.nlm.nih.gov/pubmed/38049919
http://dx.doi.org/10.1186/s41512-023-00160-2
_version_ 1785154613320613888
author Nickson, David
Singmann, Henrik
Meyer, Caroline
Toro, Carla
Walasek, Lukasz
author_facet Nickson, David
Singmann, Henrik
Meyer, Caroline
Toro, Carla
Walasek, Lukasz
author_sort Nickson, David
collection PubMed
description BACKGROUND: Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depression on the basis of individual’s primary care records. The successes of these studies are undeniable, but there is a growing concern that their results may not be replicable, which could cast doubt on their clinical usefulness. METHODS: To address this issue in the present paper, we set out to reproduce and replicate the work by Nichols et al. (2018), who trained predictive models of depression among young adults using Electronic Healthcare Records. Our contribution consists of three parts. First, we attempt to replicate the methodology used by the original authors, acquiring a more up-to-date set of primary health care records to the same specification and reproducing their data processing and analysis. Second, we test models presented in the original paper on our own data, thus providing out-of-sample prediction of the predictive models. Third, we extend past work by considering several novel machine-learning approaches in an attempt to improve the predictive accuracy achieved in the original work. RESULTS: In summary, our results demonstrate that the work of Nichols et al. is largely reproducible and replicable. This was the case both for the replication of the original model and the out-of-sample replication applying NRCBM coefficients to our new EHRs data. Although alternative predictive models did not improve model performance over standard logistic regression, our results indicate that stepwise variable selection is not stable even in the case of large data sets. CONCLUSION: We discuss the challenges associated with the research on mental health and Electronic Health Records, including the need to produce interpretable and robust models. We demonstrated some potential issues associated with the reliance on EHRs, including changes in the regulations and guidelines (such as the QOF guidelines in the UK) and reliance on visits to GP as a predictor of specific disorders.
format Online
Article
Text
id pubmed-10696659
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-106966592023-12-06 Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records Nickson, David Singmann, Henrik Meyer, Caroline Toro, Carla Walasek, Lukasz Diagn Progn Res Research BACKGROUND: Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depression on the basis of individual’s primary care records. The successes of these studies are undeniable, but there is a growing concern that their results may not be replicable, which could cast doubt on their clinical usefulness. METHODS: To address this issue in the present paper, we set out to reproduce and replicate the work by Nichols et al. (2018), who trained predictive models of depression among young adults using Electronic Healthcare Records. Our contribution consists of three parts. First, we attempt to replicate the methodology used by the original authors, acquiring a more up-to-date set of primary health care records to the same specification and reproducing their data processing and analysis. Second, we test models presented in the original paper on our own data, thus providing out-of-sample prediction of the predictive models. Third, we extend past work by considering several novel machine-learning approaches in an attempt to improve the predictive accuracy achieved in the original work. RESULTS: In summary, our results demonstrate that the work of Nichols et al. is largely reproducible and replicable. This was the case both for the replication of the original model and the out-of-sample replication applying NRCBM coefficients to our new EHRs data. Although alternative predictive models did not improve model performance over standard logistic regression, our results indicate that stepwise variable selection is not stable even in the case of large data sets. CONCLUSION: We discuss the challenges associated with the research on mental health and Electronic Health Records, including the need to produce interpretable and robust models. We demonstrated some potential issues associated with the reliance on EHRs, including changes in the regulations and guidelines (such as the QOF guidelines in the UK) and reliance on visits to GP as a predictor of specific disorders. BioMed Central 2023-12-05 /pmc/articles/PMC10696659/ /pubmed/38049919 http://dx.doi.org/10.1186/s41512-023-00160-2 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Research
Nickson, David
Singmann, Henrik
Meyer, Caroline
Toro, Carla
Walasek, Lukasz
Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
title Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
title_full Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
title_fullStr Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
title_full_unstemmed Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
title_short Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
title_sort replicability and reproducibility of predictive models for diagnosis of depression among young adults using electronic health records
topic Research
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10696659/
https://www.ncbi.nlm.nih.gov/pubmed/38049919
http://dx.doi.org/10.1186/s41512-023-00160-2
work_keys_str_mv AT nicksondavid replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords
AT singmannhenrik replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords
AT meyercaroline replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords
AT torocarla replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords
AT walaseklukasz replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords