Cargando…

Benchmarking AutoML frameworks for disease prediction using medical claims

OBJECTIVES: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. MATERIALS AND METHODS: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for di...

Descripción completa

Detalles Bibliográficos
Autores principales: A. Romero, Roland Albert, Y. Deypalan, Mariefel Nicole, Mehrotra, Suchit, Jungao, John Titus, Sheils, Natalie E., Manduchi, Elisabetta, Moore, Jason H.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9327416/
https://www.ncbi.nlm.nih.gov/pubmed/35883154
http://dx.doi.org/10.1186/s13040-022-00300-2
_version_ 1784757502133403648
author A. Romero, Roland Albert
Y. Deypalan, Mariefel Nicole
Mehrotra, Suchit
Jungao, John Titus
Sheils, Natalie E.
Manduchi, Elisabetta
Moore, Jason H.
author_facet A. Romero, Roland Albert
Y. Deypalan, Mariefel Nicole
Mehrotra, Suchit
Jungao, John Titus
Sheils, Natalie E.
Manduchi, Elisabetta
Moore, Jason H.
author_sort A. Romero, Roland Albert
collection PubMed
description OBJECTIVES: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. MATERIALS AND METHODS: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. RESULTS: The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications. DISCUSSION: Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance. CONCLUSION: Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at (10.1186/s13040-022-00300-2).
format Online
Article
Text
id pubmed-9327416
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-93274162022-07-28 Benchmarking AutoML frameworks for disease prediction using medical claims A. Romero, Roland Albert Y. Deypalan, Mariefel Nicole Mehrotra, Suchit Jungao, John Titus Sheils, Natalie E. Manduchi, Elisabetta Moore, Jason H. BioData Min Short Report OBJECTIVES: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. MATERIALS AND METHODS: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. RESULTS: The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications. DISCUSSION: Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance. CONCLUSION: Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at (10.1186/s13040-022-00300-2). BioMed Central 2022-07-26 /pmc/articles/PMC9327416/ /pubmed/35883154 http://dx.doi.org/10.1186/s13040-022-00300-2 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Short Report
A. Romero, Roland Albert
Y. Deypalan, Mariefel Nicole
Mehrotra, Suchit
Jungao, John Titus
Sheils, Natalie E.
Manduchi, Elisabetta
Moore, Jason H.
Benchmarking AutoML frameworks for disease prediction using medical claims
title Benchmarking AutoML frameworks for disease prediction using medical claims
title_full Benchmarking AutoML frameworks for disease prediction using medical claims
title_fullStr Benchmarking AutoML frameworks for disease prediction using medical claims
title_full_unstemmed Benchmarking AutoML frameworks for disease prediction using medical claims
title_short Benchmarking AutoML frameworks for disease prediction using medical claims
title_sort benchmarking automl frameworks for disease prediction using medical claims
topic Short Report
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9327416/
https://www.ncbi.nlm.nih.gov/pubmed/35883154
http://dx.doi.org/10.1186/s13040-022-00300-2
work_keys_str_mv AT aromerorolandalbert benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims
AT ydeypalanmariefelnicole benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims
AT mehrotrasuchit benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims
AT jungaojohntitus benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims
AT sheilsnataliee benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims
AT manduchielisabetta benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims
AT moorejasonh benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims