Cargando…
Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations
Machine learning methods are widely used within the medical field. However, the reliability and efficacy of these models is difficult to assess, making it difficult for researchers to identify which machine-learning model to apply to their dataset. We assessed whether variance calculations of model...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9949629/ https://www.ncbi.nlm.nih.gov/pubmed/36821544 http://dx.doi.org/10.1371/journal.pone.0281922 |
_version_ | 1784892985536675840 |
---|---|
author | Huang, Alexander A. Huang, Samuel Y. |
author_facet | Huang, Alexander A. Huang, Samuel Y. |
author_sort | Huang, Alexander A. |
collection | PubMed |
description | Machine learning methods are widely used within the medical field. However, the reliability and efficacy of these models is difficult to assess, making it difficult for researchers to identify which machine-learning model to apply to their dataset. We assessed whether variance calculations of model metrics (e.g., AUROC, Sensitivity, Specificity) through bootstrap simulation and SHapely Additive exPlanations (SHAP) could increase model transparency and improve model selection. Data from the England National Health Services Heart Disease Prediction Cohort was used. After comparison of model metrics for XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boosting, XGBoost was used as the machine-learning model of choice in this study. Boost-strap simulation (N = 10,000) was used to empirically derive the distribution of model metrics and covariate Gain statistics. SHapely Additive exPlanations (SHAP) to provide explanations to machine-learning output and simulation to evaluate the variance of model accuracy metrics. For the XGBoost modeling method, we observed (through 10,000 completed simulations) that the AUROC ranged from 0.771 to 0.947, a difference of 0.176, the balanced accuracy ranged from 0.688 to 0.894, a 0.205 difference, the sensitivity ranged from 0.632 to 0.939, a 0.307 difference, and the specificity ranged from 0.595 to 0.944, a 0.394 difference. Among 10,000 simulations completed, we observed that the gain for Angina ranged from 0.225 to 0.456, a difference of 0.231, for Cholesterol ranged from 0.148 to 0.326, a difference of 0.178, for maximum heart rate (MaxHR) ranged from 0.081 to 0.200, a range of 0.119, and for Age ranged from 0.059 to 0.157, difference of 0.098. Use of simulations to empirically evaluate the variability of model metrics and explanatory algorithms to observe if covariates match the literature are necessary for increased transparency, reliability, and utility of machine learning methods. These variance statistics, combined with model accuracy statistics can help researchers identify the best model for a given dataset. |
format | Online Article Text |
id | pubmed-9949629 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-99496292023-02-24 Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations Huang, Alexander A. Huang, Samuel Y. PLoS One Research Article Machine learning methods are widely used within the medical field. However, the reliability and efficacy of these models is difficult to assess, making it difficult for researchers to identify which machine-learning model to apply to their dataset. We assessed whether variance calculations of model metrics (e.g., AUROC, Sensitivity, Specificity) through bootstrap simulation and SHapely Additive exPlanations (SHAP) could increase model transparency and improve model selection. Data from the England National Health Services Heart Disease Prediction Cohort was used. After comparison of model metrics for XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boosting, XGBoost was used as the machine-learning model of choice in this study. Boost-strap simulation (N = 10,000) was used to empirically derive the distribution of model metrics and covariate Gain statistics. SHapely Additive exPlanations (SHAP) to provide explanations to machine-learning output and simulation to evaluate the variance of model accuracy metrics. For the XGBoost modeling method, we observed (through 10,000 completed simulations) that the AUROC ranged from 0.771 to 0.947, a difference of 0.176, the balanced accuracy ranged from 0.688 to 0.894, a 0.205 difference, the sensitivity ranged from 0.632 to 0.939, a 0.307 difference, and the specificity ranged from 0.595 to 0.944, a 0.394 difference. Among 10,000 simulations completed, we observed that the gain for Angina ranged from 0.225 to 0.456, a difference of 0.231, for Cholesterol ranged from 0.148 to 0.326, a difference of 0.178, for maximum heart rate (MaxHR) ranged from 0.081 to 0.200, a range of 0.119, and for Age ranged from 0.059 to 0.157, difference of 0.098. Use of simulations to empirically evaluate the variability of model metrics and explanatory algorithms to observe if covariates match the literature are necessary for increased transparency, reliability, and utility of machine learning methods. These variance statistics, combined with model accuracy statistics can help researchers identify the best model for a given dataset. Public Library of Science 2023-02-23 /pmc/articles/PMC9949629/ /pubmed/36821544 http://dx.doi.org/10.1371/journal.pone.0281922 Text en © 2023 Huang, Huang https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Huang, Alexander A. Huang, Samuel Y. Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
title | Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
title_full | Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
title_fullStr | Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
title_full_unstemmed | Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
title_short | Increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
title_sort | increasing transparency in machine learning through bootstrap simulation and shapely additive explanations |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9949629/ https://www.ncbi.nlm.nih.gov/pubmed/36821544 http://dx.doi.org/10.1371/journal.pone.0281922 |
work_keys_str_mv | AT huangalexandera increasingtransparencyinmachinelearningthroughbootstrapsimulationandshapelyadditiveexplanations AT huangsamuely increasingtransparencyinmachinelearningthroughbootstrapsimulationandshapelyadditiveexplanations |