Cargando…

Artificial intelligence-based methods for fusion of electronic health records and imaging data

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. The most important question when usi...

Descripción completa

Detalles Bibliográficos
Autores principales: Mohsen, Farida, Ali, Hazrat, El Hajj, Nady, Shah, Zubair
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Nature Publishing Group UK 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9605975/
https://www.ncbi.nlm.nih.gov/pubmed/36289266
http://dx.doi.org/10.1038/s41598-022-22514-4
_version_ 1784818195546243072
author Mohsen, Farida
Ali, Hazrat
El Hajj, Nady
Shah, Zubair
author_facet Mohsen, Farida
Ali, Hazrat
El Hajj, Nady
Shah, Zubair
author_sort Mohsen, Farida
collection PubMed
description Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. The most important question when using multimodal data is how to fuse them—a field of growing interest among researchers. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. After pre-processing and screening, we extracted data from 34 studies that fulfilled the inclusion criteria. We found that studies fusing imaging data with EHR are increasing and doubling from 2020 to 2021. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective. Neurological disorders were the dominant category (16 studies). From an AI perspective, conventional ML models were the most used (19 studies), followed by DL models (16 studies). Multimodal data used in the included studies were mostly from private repositories (21 studies). Through this scoping review, we offer new insights for researchers interested in knowing the current state of knowledge within this research field.
format Online
Article
Text
id pubmed-9605975
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Nature Publishing Group UK
record_format MEDLINE/PubMed
spelling pubmed-96059752022-10-28 Artificial intelligence-based methods for fusion of electronic health records and imaging data Mohsen, Farida Ali, Hazrat El Hajj, Nady Shah, Zubair Sci Rep Article Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data. Combining these multimodal data sources contributes to a better understanding of human health and provides optimal personalized healthcare. The most important question when using multimodal data is how to fuse them—a field of growing interest among researchers. Advances in artificial intelligence (AI) technologies, particularly machine learning (ML), enable the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical applications. More specifically, we focus on studies that only fused EHR with medical imaging data to develop various AI methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical application, and the available multimodal medical datasets. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched Embase, PubMed, Scopus, and Google Scholar to retrieve relevant studies. After pre-processing and screening, we extracted data from 34 studies that fulfilled the inclusion criteria. We found that studies fusing imaging data with EHR are increasing and doubling from 2020 to 2021. In our analysis, a typical workflow was observed: feeding raw data, fusing different data modalities by applying conventional machine learning (ML) or deep learning (DL) algorithms, and finally, evaluating the multimodal fusion through clinical outcome predictions. Specifically, early fusion was the most used technique in most applications for multimodal learning (22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively) from a clinical outcome perspective. Neurological disorders were the dominant category (16 studies). From an AI perspective, conventional ML models were the most used (19 studies), followed by DL models (16 studies). Multimodal data used in the included studies were mostly from private repositories (21 studies). Through this scoping review, we offer new insights for researchers interested in knowing the current state of knowledge within this research field. Nature Publishing Group UK 2022-10-26 /pmc/articles/PMC9605975/ /pubmed/36289266 http://dx.doi.org/10.1038/s41598-022-22514-4 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Article
Mohsen, Farida
Ali, Hazrat
El Hajj, Nady
Shah, Zubair
Artificial intelligence-based methods for fusion of electronic health records and imaging data
title Artificial intelligence-based methods for fusion of electronic health records and imaging data
title_full Artificial intelligence-based methods for fusion of electronic health records and imaging data
title_fullStr Artificial intelligence-based methods for fusion of electronic health records and imaging data
title_full_unstemmed Artificial intelligence-based methods for fusion of electronic health records and imaging data
title_short Artificial intelligence-based methods for fusion of electronic health records and imaging data
title_sort artificial intelligence-based methods for fusion of electronic health records and imaging data
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9605975/
https://www.ncbi.nlm.nih.gov/pubmed/36289266
http://dx.doi.org/10.1038/s41598-022-22514-4
work_keys_str_mv AT mohsenfarida artificialintelligencebasedmethodsforfusionofelectronichealthrecordsandimagingdata
AT alihazrat artificialintelligencebasedmethodsforfusionofelectronichealthrecordsandimagingdata
AT elhajjnady artificialintelligencebasedmethodsforfusionofelectronichealthrecordsandimagingdata
AT shahzubair artificialintelligencebasedmethodsforfusionofelectronichealthrecordsandimagingdata