Cargando…
Benchmarking AutoML frameworks for disease prediction using medical claims
OBJECTIVES: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. MATERIALS AND METHODS: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for di...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9327416/ https://www.ncbi.nlm.nih.gov/pubmed/35883154 http://dx.doi.org/10.1186/s13040-022-00300-2 |
_version_ | 1784757502133403648 |
---|---|
author | A. Romero, Roland Albert Y. Deypalan, Mariefel Nicole Mehrotra, Suchit Jungao, John Titus Sheils, Natalie E. Manduchi, Elisabetta Moore, Jason H. |
author_facet | A. Romero, Roland Albert Y. Deypalan, Mariefel Nicole Mehrotra, Suchit Jungao, John Titus Sheils, Natalie E. Manduchi, Elisabetta Moore, Jason H. |
author_sort | A. Romero, Roland Albert |
collection | PubMed |
description | OBJECTIVES: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. MATERIALS AND METHODS: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. RESULTS: The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications. DISCUSSION: Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance. CONCLUSION: Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at (10.1186/s13040-022-00300-2). |
format | Online Article Text |
id | pubmed-9327416 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-93274162022-07-28 Benchmarking AutoML frameworks for disease prediction using medical claims A. Romero, Roland Albert Y. Deypalan, Mariefel Nicole Mehrotra, Suchit Jungao, John Titus Sheils, Natalie E. Manduchi, Elisabetta Moore, Jason H. BioData Min Short Report OBJECTIVES: Ascertain and compare the performances of Automated Machine Learning (AutoML) tools on large, highly imbalanced healthcare datasets. MATERIALS AND METHODS: We generated a large dataset using historical de-identified administrative claims including demographic information and flags for disease codes in four different time windows prior to 2019. We then trained three AutoML tools on this dataset to predict six different disease outcomes in 2019 and evaluated model performances on several metrics. RESULTS: The AutoML tools showed improvement from the baseline random forest model but did not differ significantly from each other. All models recorded low area under the precision-recall curve and failed to predict true positives while keeping the true negative rate high. Model performance was not directly related to prevalence. We provide a specific use-case to illustrate how to select a threshold that gives the best balance between true and false positive rates, as this is an important consideration in medical applications. DISCUSSION: Healthcare datasets present several challenges for AutoML tools, including large sample size, high imbalance, and limitations in the available features. Improvements in scalability, combinations of imbalance-learning resampling and ensemble approaches, and curated feature selection are possible next steps to achieve better performance. CONCLUSION: Among the three explored, no AutoML tool consistently outperforms the rest in terms of predictive performance. The performances of the models in this study suggest that there may be room for improvement in handling medical claims data. Finally, selection of the optimal prediction threshold should be guided by the specific practical application. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at (10.1186/s13040-022-00300-2). BioMed Central 2022-07-26 /pmc/articles/PMC9327416/ /pubmed/35883154 http://dx.doi.org/10.1186/s13040-022-00300-2 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
spellingShingle | Short Report A. Romero, Roland Albert Y. Deypalan, Mariefel Nicole Mehrotra, Suchit Jungao, John Titus Sheils, Natalie E. Manduchi, Elisabetta Moore, Jason H. Benchmarking AutoML frameworks for disease prediction using medical claims |
title | Benchmarking AutoML frameworks for disease prediction using medical claims |
title_full | Benchmarking AutoML frameworks for disease prediction using medical claims |
title_fullStr | Benchmarking AutoML frameworks for disease prediction using medical claims |
title_full_unstemmed | Benchmarking AutoML frameworks for disease prediction using medical claims |
title_short | Benchmarking AutoML frameworks for disease prediction using medical claims |
title_sort | benchmarking automl frameworks for disease prediction using medical claims |
topic | Short Report |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9327416/ https://www.ncbi.nlm.nih.gov/pubmed/35883154 http://dx.doi.org/10.1186/s13040-022-00300-2 |
work_keys_str_mv | AT aromerorolandalbert benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims AT ydeypalanmariefelnicole benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims AT mehrotrasuchit benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims AT jungaojohntitus benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims AT sheilsnataliee benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims AT manduchielisabetta benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims AT moorejasonh benchmarkingautomlframeworksfordiseasepredictionusingmedicalclaims |