Cargando…
Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summ...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10030980/ https://www.ncbi.nlm.nih.gov/pubmed/36968273 http://dx.doi.org/10.3389/fmolb.2023.1136071 |
_version_ | 1784910498741878784 |
---|---|
author | Niu, Ke Zhang, Ke Peng, Xueping Pan, Yijie Xiao, Naian |
author_facet | Niu, Ke Zhang, Ke Peng, Xueping Pan, Yijie Xiao, Naian |
author_sort | Niu, Ke |
collection | PubMed |
description | In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients’ time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient’s clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient’s time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline. |
format | Online Article Text |
id | pubmed-10030980 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-100309802023-03-23 Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction Niu, Ke Zhang, Ke Peng, Xueping Pan, Yijie Xiao, Naian Front Mol Biosci Molecular Biosciences In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients’ time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient’s clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient’s time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline. Frontiers Media S.A. 2023-03-08 /pmc/articles/PMC10030980/ /pubmed/36968273 http://dx.doi.org/10.3389/fmolb.2023.1136071 Text en Copyright © 2023 Niu, Zhang, Peng, Pan and Xiao. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Molecular Biosciences Niu, Ke Zhang, Ke Peng, Xueping Pan, Yijie Xiao, Naian Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
title | Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
title_full | Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
title_fullStr | Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
title_full_unstemmed | Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
title_short | Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
title_sort | deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction |
topic | Molecular Biosciences |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10030980/ https://www.ncbi.nlm.nih.gov/pubmed/36968273 http://dx.doi.org/10.3389/fmolb.2023.1136071 |
work_keys_str_mv | AT niuke deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction AT zhangke deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction AT pengxueping deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction AT panyijie deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction AT xiaonaian deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction |