Cargando…
Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis
BACKGROUND: Asthma exerts a substantial burden on patients and health care systems. To facilitate preventive care for asthma management and improve patient outcomes, we recently developed two machine learning models, one on Intermountain Healthcare data and the other on Kaiser Permanente Southern Ca...
Autores principales: | , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
JMIR Publications
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8085752/ https://www.ncbi.nlm.nih.gov/pubmed/33856359 http://dx.doi.org/10.2196/24153 |
_version_ | 1783686410511843328 |
---|---|
author | Luo, Gang Nau, Claudia L Crawford, William W Schatz, Michael Zeiger, Robert S Koebnick, Corinna |
author_facet | Luo, Gang Nau, Claudia L Crawford, William W Schatz, Michael Zeiger, Robert S Koebnick, Corinna |
author_sort | Luo, Gang |
collection | PubMed |
description | BACKGROUND: Asthma exerts a substantial burden on patients and health care systems. To facilitate preventive care for asthma management and improve patient outcomes, we recently developed two machine learning models, one on Intermountain Healthcare data and the other on Kaiser Permanente Southern California (KPSC) data, to forecast asthma-related hospital visits, including emergency department visits and hospitalizations, in the succeeding 12 months among patients with asthma. As is typical for machine learning approaches, these two models do not explain their forecasting results. To address the interpretability issue of black-box models, we designed an automatic method to offer rule format explanations for the forecasting results of any machine learning model on imbalanced tabular data and to suggest customized interventions with no accuracy loss. Our method worked well for explaining the forecasting results of our Intermountain Healthcare model, but its generalizability to other health care systems remains unknown. OBJECTIVE: The objective of this study is to evaluate the generalizability of our automatic explanation method to KPSC for forecasting asthma-related hospital visits. METHODS: Through a secondary analysis of 987,506 data instances from 2012 to 2017 at KPSC, we used our method to explain the forecasting results of our KPSC model and to suggest customized interventions. The patient cohort covered a random sample of 70% of patients with asthma who had a KPSC health plan for any period between 2015 and 2018. RESULTS: Our method explained the forecasting results for 97.57% (2204/2259) of the patients with asthma who were correctly forecasted to undergo asthma-related hospital visits in the succeeding 12 months. CONCLUSIONS: For forecasting asthma-related hospital visits, our automatic explanation method exhibited an acceptable generalizability to KPSC. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/resprot.5039 |
format | Online Article Text |
id | pubmed-8085752 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | JMIR Publications |
record_format | MEDLINE/PubMed |
spelling | pubmed-80857522021-05-06 Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis Luo, Gang Nau, Claudia L Crawford, William W Schatz, Michael Zeiger, Robert S Koebnick, Corinna J Med Internet Res Original Paper BACKGROUND: Asthma exerts a substantial burden on patients and health care systems. To facilitate preventive care for asthma management and improve patient outcomes, we recently developed two machine learning models, one on Intermountain Healthcare data and the other on Kaiser Permanente Southern California (KPSC) data, to forecast asthma-related hospital visits, including emergency department visits and hospitalizations, in the succeeding 12 months among patients with asthma. As is typical for machine learning approaches, these two models do not explain their forecasting results. To address the interpretability issue of black-box models, we designed an automatic method to offer rule format explanations for the forecasting results of any machine learning model on imbalanced tabular data and to suggest customized interventions with no accuracy loss. Our method worked well for explaining the forecasting results of our Intermountain Healthcare model, but its generalizability to other health care systems remains unknown. OBJECTIVE: The objective of this study is to evaluate the generalizability of our automatic explanation method to KPSC for forecasting asthma-related hospital visits. METHODS: Through a secondary analysis of 987,506 data instances from 2012 to 2017 at KPSC, we used our method to explain the forecasting results of our KPSC model and to suggest customized interventions. The patient cohort covered a random sample of 70% of patients with asthma who had a KPSC health plan for any period between 2015 and 2018. RESULTS: Our method explained the forecasting results for 97.57% (2204/2259) of the patients with asthma who were correctly forecasted to undergo asthma-related hospital visits in the succeeding 12 months. CONCLUSIONS: For forecasting asthma-related hospital visits, our automatic explanation method exhibited an acceptable generalizability to KPSC. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/resprot.5039 JMIR Publications 2021-04-15 /pmc/articles/PMC8085752/ /pubmed/33856359 http://dx.doi.org/10.2196/24153 Text en ©Gang Luo, Claudia L Nau, William W Crawford, Michael Schatz, Robert S Zeiger, Corinna Koebnick. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 15.04.2021. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included. |
spellingShingle | Original Paper Luo, Gang Nau, Claudia L Crawford, William W Schatz, Michael Zeiger, Robert S Koebnick, Corinna Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis |
title | Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis |
title_full | Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis |
title_fullStr | Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis |
title_full_unstemmed | Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis |
title_short | Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis |
title_sort | generalizability of an automatic explanation method for machine learning prediction results on asthma-related hospital visits in patients with asthma: quantitative analysis |
topic | Original Paper |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8085752/ https://www.ncbi.nlm.nih.gov/pubmed/33856359 http://dx.doi.org/10.2196/24153 |
work_keys_str_mv | AT luogang generalizabilityofanautomaticexplanationmethodformachinelearningpredictionresultsonasthmarelatedhospitalvisitsinpatientswithasthmaquantitativeanalysis AT nauclaudial generalizabilityofanautomaticexplanationmethodformachinelearningpredictionresultsonasthmarelatedhospitalvisitsinpatientswithasthmaquantitativeanalysis AT crawfordwilliamw generalizabilityofanautomaticexplanationmethodformachinelearningpredictionresultsonasthmarelatedhospitalvisitsinpatientswithasthmaquantitativeanalysis AT schatzmichael generalizabilityofanautomaticexplanationmethodformachinelearningpredictionresultsonasthmarelatedhospitalvisitsinpatientswithasthmaquantitativeanalysis AT zeigerroberts generalizabilityofanautomaticexplanationmethodformachinelearningpredictionresultsonasthmarelatedhospitalvisitsinpatientswithasthmaquantitativeanalysis AT koebnickcorinna generalizabilityofanautomaticexplanationmethodformachinelearningpredictionresultsonasthmarelatedhospitalvisitsinpatientswithasthmaquantitativeanalysis |