Cargando…
Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records
BACKGROUND: Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depress...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10696659/ https://www.ncbi.nlm.nih.gov/pubmed/38049919 http://dx.doi.org/10.1186/s41512-023-00160-2 |
_version_ | 1785154613320613888 |
---|---|
author | Nickson, David Singmann, Henrik Meyer, Caroline Toro, Carla Walasek, Lukasz |
author_facet | Nickson, David Singmann, Henrik Meyer, Caroline Toro, Carla Walasek, Lukasz |
author_sort | Nickson, David |
collection | PubMed |
description | BACKGROUND: Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depression on the basis of individual’s primary care records. The successes of these studies are undeniable, but there is a growing concern that their results may not be replicable, which could cast doubt on their clinical usefulness. METHODS: To address this issue in the present paper, we set out to reproduce and replicate the work by Nichols et al. (2018), who trained predictive models of depression among young adults using Electronic Healthcare Records. Our contribution consists of three parts. First, we attempt to replicate the methodology used by the original authors, acquiring a more up-to-date set of primary health care records to the same specification and reproducing their data processing and analysis. Second, we test models presented in the original paper on our own data, thus providing out-of-sample prediction of the predictive models. Third, we extend past work by considering several novel machine-learning approaches in an attempt to improve the predictive accuracy achieved in the original work. RESULTS: In summary, our results demonstrate that the work of Nichols et al. is largely reproducible and replicable. This was the case both for the replication of the original model and the out-of-sample replication applying NRCBM coefficients to our new EHRs data. Although alternative predictive models did not improve model performance over standard logistic regression, our results indicate that stepwise variable selection is not stable even in the case of large data sets. CONCLUSION: We discuss the challenges associated with the research on mental health and Electronic Health Records, including the need to produce interpretable and robust models. We demonstrated some potential issues associated with the reliance on EHRs, including changes in the regulations and guidelines (such as the QOF guidelines in the UK) and reliance on visits to GP as a predictor of specific disorders. |
format | Online Article Text |
id | pubmed-10696659 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-106966592023-12-06 Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records Nickson, David Singmann, Henrik Meyer, Caroline Toro, Carla Walasek, Lukasz Diagn Progn Res Research BACKGROUND: Recent advances in machine learning combined with the growing availability of digitized health records offer new opportunities for improving early diagnosis of depression. An emerging body of research shows that Electronic Health Records can be used to accurately predict cases of depression on the basis of individual’s primary care records. The successes of these studies are undeniable, but there is a growing concern that their results may not be replicable, which could cast doubt on their clinical usefulness. METHODS: To address this issue in the present paper, we set out to reproduce and replicate the work by Nichols et al. (2018), who trained predictive models of depression among young adults using Electronic Healthcare Records. Our contribution consists of three parts. First, we attempt to replicate the methodology used by the original authors, acquiring a more up-to-date set of primary health care records to the same specification and reproducing their data processing and analysis. Second, we test models presented in the original paper on our own data, thus providing out-of-sample prediction of the predictive models. Third, we extend past work by considering several novel machine-learning approaches in an attempt to improve the predictive accuracy achieved in the original work. RESULTS: In summary, our results demonstrate that the work of Nichols et al. is largely reproducible and replicable. This was the case both for the replication of the original model and the out-of-sample replication applying NRCBM coefficients to our new EHRs data. Although alternative predictive models did not improve model performance over standard logistic regression, our results indicate that stepwise variable selection is not stable even in the case of large data sets. CONCLUSION: We discuss the challenges associated with the research on mental health and Electronic Health Records, including the need to produce interpretable and robust models. We demonstrated some potential issues associated with the reliance on EHRs, including changes in the regulations and guidelines (such as the QOF guidelines in the UK) and reliance on visits to GP as a predictor of specific disorders. BioMed Central 2023-12-05 /pmc/articles/PMC10696659/ /pubmed/38049919 http://dx.doi.org/10.1186/s41512-023-00160-2 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Research Nickson, David Singmann, Henrik Meyer, Caroline Toro, Carla Walasek, Lukasz Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records |
title | Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records |
title_full | Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records |
title_fullStr | Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records |
title_full_unstemmed | Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records |
title_short | Replicability and reproducibility of predictive models for diagnosis of depression among young adults using Electronic Health Records |
title_sort | replicability and reproducibility of predictive models for diagnosis of depression among young adults using electronic health records |
topic | Research |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10696659/ https://www.ncbi.nlm.nih.gov/pubmed/38049919 http://dx.doi.org/10.1186/s41512-023-00160-2 |
work_keys_str_mv | AT nicksondavid replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords AT singmannhenrik replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords AT meyercaroline replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords AT torocarla replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords AT walaseklukasz replicabilityandreproducibilityofpredictivemodelsfordiagnosisofdepressionamongyoungadultsusingelectronichealthrecords |