Cargando…
Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology
Uncertainty estimation is crucial for understanding the reliability of deep learning (DL) predictions, and critical for deploying DL in the clinic. Differences between training and production datasets can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we...
Autores principales: | , , , , , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Nature Publishing Group UK
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10164181/ https://www.ncbi.nlm.nih.gov/pubmed/37149669 http://dx.doi.org/10.1038/s41598-023-31126-5 |
_version_ | 1785038017232109568 |
---|---|
author | MacDonald, Samual Foley, Helena Yap, Melvyn Johnston, Rebecca L. Steven, Kaiah Koufariotis, Lambros T. Sharma, Sowmya Wood, Scott Addala, Venkateswar Pearson, John V. Roosta, Fred Waddell, Nicola Kondrashova, Olga Trzaskowski, Maciej |
author_facet | MacDonald, Samual Foley, Helena Yap, Melvyn Johnston, Rebecca L. Steven, Kaiah Koufariotis, Lambros T. Sharma, Sowmya Wood, Scott Addala, Venkateswar Pearson, John V. Roosta, Fred Waddell, Nicola Kondrashova, Olga Trzaskowski, Maciej |
author_sort | MacDonald, Samual |
collection | PubMed |
description | Uncertainty estimation is crucial for understanding the reliability of deep learning (DL) predictions, and critical for deploying DL in the clinic. Differences between training and production datasets can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models for predicting cancer of unknown primary, using three RNA-seq datasets with 10,968 samples across 57 cancer types. Our results highlight that simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation. Moreover, we designed a prototypical metric—the area between development and production curve (ADP), which evaluates the accuracy loss when deploying models from development to production. Using ADP, we demonstrate that Bayesian DL improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’. In summary, Bayesian DL is a promising approach for generalising uncertainty, improving performance, transparency, and safety of DL models for deployment in the real world. |
format | Online Article Text |
id | pubmed-10164181 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Nature Publishing Group UK |
record_format | MEDLINE/PubMed |
spelling | pubmed-101641812023-05-08 Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology MacDonald, Samual Foley, Helena Yap, Melvyn Johnston, Rebecca L. Steven, Kaiah Koufariotis, Lambros T. Sharma, Sowmya Wood, Scott Addala, Venkateswar Pearson, John V. Roosta, Fred Waddell, Nicola Kondrashova, Olga Trzaskowski, Maciej Sci Rep Article Uncertainty estimation is crucial for understanding the reliability of deep learning (DL) predictions, and critical for deploying DL in the clinic. Differences between training and production datasets can lead to incorrect predictions with underestimated uncertainty. To investigate this pitfall, we benchmarked one pointwise and three approximate Bayesian DL models for predicting cancer of unknown primary, using three RNA-seq datasets with 10,968 samples across 57 cancer types. Our results highlight that simple and scalable Bayesian DL significantly improves the generalisation of uncertainty estimation. Moreover, we designed a prototypical metric—the area between development and production curve (ADP), which evaluates the accuracy loss when deploying models from development to production. Using ADP, we demonstrate that Bayesian DL improves accuracy under data distributional shifts when utilising ‘uncertainty thresholding’. In summary, Bayesian DL is a promising approach for generalising uncertainty, improving performance, transparency, and safety of DL models for deployment in the real world. Nature Publishing Group UK 2023-05-06 /pmc/articles/PMC10164181/ /pubmed/37149669 http://dx.doi.org/10.1038/s41598-023-31126-5 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by/4.0/Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Article MacDonald, Samual Foley, Helena Yap, Melvyn Johnston, Rebecca L. Steven, Kaiah Koufariotis, Lambros T. Sharma, Sowmya Wood, Scott Addala, Venkateswar Pearson, John V. Roosta, Fred Waddell, Nicola Kondrashova, Olga Trzaskowski, Maciej Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
title | Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
title_full | Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
title_fullStr | Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
title_full_unstemmed | Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
title_short | Generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
title_sort | generalising uncertainty improves accuracy and safety of deep learning analytics applied to oncology |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10164181/ https://www.ncbi.nlm.nih.gov/pubmed/37149669 http://dx.doi.org/10.1038/s41598-023-31126-5 |
work_keys_str_mv | AT macdonaldsamual generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT foleyhelena generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT yapmelvyn generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT johnstonrebeccal generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT stevenkaiah generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT koufariotislambrost generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT sharmasowmya generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT woodscott generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT addalavenkateswar generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT pearsonjohnv generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT roostafred generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT waddellnicola generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT kondrashovaolga generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology AT trzaskowskimaciej generalisinguncertaintyimprovesaccuracyandsafetyofdeeplearninganalyticsappliedtooncology |