Cargando…

Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction

In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summ...

Descripción completa

Detalles Bibliográficos
Autores principales: Niu, Ke, Zhang, Ke, Peng, Xueping, Pan, Yijie, Xiao, Naian
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10030980/
https://www.ncbi.nlm.nih.gov/pubmed/36968273
http://dx.doi.org/10.3389/fmolb.2023.1136071
_version_ 1784910498741878784
author Niu, Ke
Zhang, Ke
Peng, Xueping
Pan, Yijie
Xiao, Naian
author_facet Niu, Ke
Zhang, Ke
Peng, Xueping
Pan, Yijie
Xiao, Naian
author_sort Niu, Ke
collection PubMed
description In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients’ time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient’s clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient’s time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline.
format Online
Article
Text
id pubmed-10030980
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-100309802023-03-23 Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction Niu, Ke Zhang, Ke Peng, Xueping Pan, Yijie Xiao, Naian Front Mol Biosci Molecular Biosciences In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients’ time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient’s clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient’s time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline. Frontiers Media S.A. 2023-03-08 /pmc/articles/PMC10030980/ /pubmed/36968273 http://dx.doi.org/10.3389/fmolb.2023.1136071 Text en Copyright © 2023 Niu, Zhang, Peng, Pan and Xiao. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Molecular Biosciences
Niu, Ke
Zhang, Ke
Peng, Xueping
Pan, Yijie
Xiao, Naian
Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
title Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
title_full Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
title_fullStr Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
title_full_unstemmed Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
title_short Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
title_sort deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction
topic Molecular Biosciences
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10030980/
https://www.ncbi.nlm.nih.gov/pubmed/36968273
http://dx.doi.org/10.3389/fmolb.2023.1136071
work_keys_str_mv AT niuke deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction
AT zhangke deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction
AT pengxueping deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction
AT panyijie deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction
AT xiaonaian deepmultimodalintermediatefusionofclinicalrecordandtimeseriesdatainmortalityprediction