Cargando…
Uncertainty-aware deep learning in healthcare: A scoping review
Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there i...
Autores principales: | , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9802673/ https://www.ncbi.nlm.nih.gov/pubmed/36590140 http://dx.doi.org/10.1371/journal.pdig.0000085 |
_version_ | 1784861723837071360 |
---|---|
author | Loftus, Tyler J. Shickel, Benjamin Ruppert, Matthew M. Balch, Jeremy A. Ozrazgat-Baslanti, Tezcan Tighe, Patrick J. Efron, Philip A. Hogan, William R. Rashidi, Parisa Upchurch, Gilbert R. Bihorac, Azra |
author_facet | Loftus, Tyler J. Shickel, Benjamin Ruppert, Matthew M. Balch, Jeremy A. Ozrazgat-Baslanti, Tezcan Tighe, Patrick J. Efron, Philip A. Hogan, William R. Rashidi, Parisa Upchurch, Gilbert R. Bihorac, Azra |
author_sort | Loftus, Tyler J. |
collection | PubMed |
description | Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there is no consensus regarding optimal methods for quantifying uncertainty. Our purpose is to critically evaluate methods for quantifying uncertainty in deep learning for healthcare applications and propose a conceptual framework for specifying certainty of deep learning predictions. We searched Embase, MEDLINE, and PubMed databases for articles relevant to study objectives, complying with PRISMA guidelines, rated study quality using validated tools, and extracted data according to modified CHARMS criteria. Among 30 included studies, 24 described medical imaging applications. All imaging model architectures used convolutional neural networks or a variation thereof. The predominant method for quantifying uncertainty was Monte Carlo dropout, producing predictions from multiple networks for which different neurons have dropped out and measuring variance across the distribution of resulting predictions. Conformal prediction offered similar strong performance in estimating uncertainty, along with ease of interpretation and application not only to deep learning but also to other machine learning approaches. Among the six articles describing non-imaging applications, model architectures and uncertainty estimation methods were heterogeneous, but predictive performance was generally strong, and uncertainty estimation was effective in comparing modeling methods. Overall, the use of model learning curves to quantify epistemic uncertainty (attributable to model parameters) was sparse. Heterogeneity in reporting methods precluded the performance of a meta-analysis. Uncertainty estimation methods have the potential to identify rare but important misclassifications made by deep learning models and compare modeling methods, which could build patient and clinician trust in deep learning applications in healthcare. Efficient maturation of this field will require standardized guidelines for reporting performance and uncertainty metrics. |
format | Online Article Text |
id | pubmed-9802673 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-98026732022-12-30 Uncertainty-aware deep learning in healthcare: A scoping review Loftus, Tyler J. Shickel, Benjamin Ruppert, Matthew M. Balch, Jeremy A. Ozrazgat-Baslanti, Tezcan Tighe, Patrick J. Efron, Philip A. Hogan, William R. Rashidi, Parisa Upchurch, Gilbert R. Bihorac, Azra PLOS Digit Health Research Article Mistrust is a major barrier to implementing deep learning in healthcare settings. Entrustment could be earned by conveying model certainty, or the probability that a given model output is accurate, but the use of uncertainty estimation for deep learning entrustment is largely unexplored, and there is no consensus regarding optimal methods for quantifying uncertainty. Our purpose is to critically evaluate methods for quantifying uncertainty in deep learning for healthcare applications and propose a conceptual framework for specifying certainty of deep learning predictions. We searched Embase, MEDLINE, and PubMed databases for articles relevant to study objectives, complying with PRISMA guidelines, rated study quality using validated tools, and extracted data according to modified CHARMS criteria. Among 30 included studies, 24 described medical imaging applications. All imaging model architectures used convolutional neural networks or a variation thereof. The predominant method for quantifying uncertainty was Monte Carlo dropout, producing predictions from multiple networks for which different neurons have dropped out and measuring variance across the distribution of resulting predictions. Conformal prediction offered similar strong performance in estimating uncertainty, along with ease of interpretation and application not only to deep learning but also to other machine learning approaches. Among the six articles describing non-imaging applications, model architectures and uncertainty estimation methods were heterogeneous, but predictive performance was generally strong, and uncertainty estimation was effective in comparing modeling methods. Overall, the use of model learning curves to quantify epistemic uncertainty (attributable to model parameters) was sparse. Heterogeneity in reporting methods precluded the performance of a meta-analysis. Uncertainty estimation methods have the potential to identify rare but important misclassifications made by deep learning models and compare modeling methods, which could build patient and clinician trust in deep learning applications in healthcare. Efficient maturation of this field will require standardized guidelines for reporting performance and uncertainty metrics. Public Library of Science 2022-08-10 /pmc/articles/PMC9802673/ /pubmed/36590140 http://dx.doi.org/10.1371/journal.pdig.0000085 Text en © 2022 Loftus et al https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. |
spellingShingle | Research Article Loftus, Tyler J. Shickel, Benjamin Ruppert, Matthew M. Balch, Jeremy A. Ozrazgat-Baslanti, Tezcan Tighe, Patrick J. Efron, Philip A. Hogan, William R. Rashidi, Parisa Upchurch, Gilbert R. Bihorac, Azra Uncertainty-aware deep learning in healthcare: A scoping review |
title | Uncertainty-aware deep learning in healthcare: A scoping review |
title_full | Uncertainty-aware deep learning in healthcare: A scoping review |
title_fullStr | Uncertainty-aware deep learning in healthcare: A scoping review |
title_full_unstemmed | Uncertainty-aware deep learning in healthcare: A scoping review |
title_short | Uncertainty-aware deep learning in healthcare: A scoping review |
title_sort | uncertainty-aware deep learning in healthcare: a scoping review |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9802673/ https://www.ncbi.nlm.nih.gov/pubmed/36590140 http://dx.doi.org/10.1371/journal.pdig.0000085 |
work_keys_str_mv | AT loftustylerj uncertaintyawaredeeplearninginhealthcareascopingreview AT shickelbenjamin uncertaintyawaredeeplearninginhealthcareascopingreview AT ruppertmatthewm uncertaintyawaredeeplearninginhealthcareascopingreview AT balchjeremya uncertaintyawaredeeplearninginhealthcareascopingreview AT ozrazgatbaslantitezcan uncertaintyawaredeeplearninginhealthcareascopingreview AT tighepatrickj uncertaintyawaredeeplearninginhealthcareascopingreview AT efronphilipa uncertaintyawaredeeplearninginhealthcareascopingreview AT hoganwilliamr uncertaintyawaredeeplearninginhealthcareascopingreview AT rashidiparisa uncertaintyawaredeeplearninginhealthcareascopingreview AT upchurchgilbertr uncertaintyawaredeeplearninginhealthcareascopingreview AT bihoracazra uncertaintyawaredeeplearninginhealthcareascopingreview |