Cargando…
Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection
Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to int...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Nature Publishing Group UK
2020
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7746687/ https://www.ncbi.nlm.nih.gov/pubmed/33335111 http://dx.doi.org/10.1038/s41598-020-78888-w |
_version_ | 1783624841643950080 |
---|---|
author | Huang, Shih-Cheng Pareek, Anuj Zamanian, Roham Banerjee, Imon Lungren, Matthew P. |
author_facet | Huang, Shih-Cheng Pareek, Anuj Zamanian, Roham Banerjee, Imon Lungren, Matthew P. |
author_sort | Huang, Shih-Cheng |
collection | PubMed |
description | Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to integrate both clinical and imaging data, despite that in routine practice clinicians rely on EMR to provide context in medical imaging interpretation. In this study, we developed and compared different multimodal fusion model architectures that are capable of utilizing both pixel data from volumetric Computed Tomography Pulmonary Angiography scans and clinical patient data from the EMR to automatically classify Pulmonary Embolism (PE) cases. The best performing multimodality model is a late fusion model that achieves an AUROC of 0.947 [95% CI: 0.946–0.948] on the entire held-out test set, outperforming imaging-only and EMR-only single modality models. |
format | Online Article Text |
id | pubmed-7746687 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2020 |
publisher | Nature Publishing Group UK |
record_format | MEDLINE/PubMed |
spelling | pubmed-77466872020-12-18 Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection Huang, Shih-Cheng Pareek, Anuj Zamanian, Roham Banerjee, Imon Lungren, Matthew P. Sci Rep Article Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to integrate both clinical and imaging data, despite that in routine practice clinicians rely on EMR to provide context in medical imaging interpretation. In this study, we developed and compared different multimodal fusion model architectures that are capable of utilizing both pixel data from volumetric Computed Tomography Pulmonary Angiography scans and clinical patient data from the EMR to automatically classify Pulmonary Embolism (PE) cases. The best performing multimodality model is a late fusion model that achieves an AUROC of 0.947 [95% CI: 0.946–0.948] on the entire held-out test set, outperforming imaging-only and EMR-only single modality models. Nature Publishing Group UK 2020-12-17 /pmc/articles/PMC7746687/ /pubmed/33335111 http://dx.doi.org/10.1038/s41598-020-78888-w Text en © The Author(s) 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
spellingShingle | Article Huang, Shih-Cheng Pareek, Anuj Zamanian, Roham Banerjee, Imon Lungren, Matthew P. Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection |
title | Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection |
title_full | Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection |
title_fullStr | Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection |
title_full_unstemmed | Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection |
title_short | Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection |
title_sort | multimodal fusion with deep neural networks for leveraging ct imaging and electronic health record: a case-study in pulmonary embolism detection |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7746687/ https://www.ncbi.nlm.nih.gov/pubmed/33335111 http://dx.doi.org/10.1038/s41598-020-78888-w |
work_keys_str_mv | AT huangshihcheng multimodalfusionwithdeepneuralnetworksforleveragingctimagingandelectronichealthrecordacasestudyinpulmonaryembolismdetection AT pareekanuj multimodalfusionwithdeepneuralnetworksforleveragingctimagingandelectronichealthrecordacasestudyinpulmonaryembolismdetection AT zamanianroham multimodalfusionwithdeepneuralnetworksforleveragingctimagingandelectronichealthrecordacasestudyinpulmonaryembolismdetection AT banerjeeimon multimodalfusionwithdeepneuralnetworksforleveragingctimagingandelectronichealthrecordacasestudyinpulmonaryembolismdetection AT lungrenmatthewp multimodalfusionwithdeepneuralnetworksforleveragingctimagingandelectronichealthrecordacasestudyinpulmonaryembolismdetection |