Cargando…

DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion

Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special lear...

Descripción completa

Detalles Bibliográficos
Autores principales: Huang, Wanwan, Zhang, Han, Cheng, Yu, Quan, Xiongwen
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10656763/
https://www.ncbi.nlm.nih.gov/pubmed/38028809
http://dx.doi.org/10.3389/fphys.2023.1241370
_version_ 1785137070610579456
author Huang, Wanwan
Zhang, Han
Cheng, Yu
Quan, Xiongwen
author_facet Huang, Wanwan
Zhang, Han
Cheng, Yu
Quan, Xiongwen
author_sort Huang, Wanwan
collection PubMed
description Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism-based disentangled representation network for medical image fusion, which designs coordinate attention and multimodal attention to extract and strengthen common and exclusive features. First, the common and exclusive features of each modality were obtained by the cross mutual information and adversarial objective methods, respectively. Then, coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, these two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis, and four metrics are used for quantitative comparison. The values of the four metrics demonstrate the effect of the DRCM. Furthermore, the DRCM achieved better results on SCD, Nabf, and MS-SSIM metrics, which indicates that the DRCM achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, it was found that the DRCM outperforms the comparison method.
format Online
Article
Text
id pubmed-10656763
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-106567632023-11-03 DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion Huang, Wanwan Zhang, Han Cheng, Yu Quan, Xiongwen Front Physiol Physiology Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism-based disentangled representation network for medical image fusion, which designs coordinate attention and multimodal attention to extract and strengthen common and exclusive features. First, the common and exclusive features of each modality were obtained by the cross mutual information and adversarial objective methods, respectively. Then, coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, these two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis, and four metrics are used for quantitative comparison. The values of the four metrics demonstrate the effect of the DRCM. Furthermore, the DRCM achieved better results on SCD, Nabf, and MS-SSIM metrics, which indicates that the DRCM achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, it was found that the DRCM outperforms the comparison method. Frontiers Media S.A. 2023-11-03 /pmc/articles/PMC10656763/ /pubmed/38028809 http://dx.doi.org/10.3389/fphys.2023.1241370 Text en Copyright © 2023 Huang, Zhang, Cheng and Quan. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Physiology
Huang, Wanwan
Zhang, Han
Cheng, Yu
Quan, Xiongwen
DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
title DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
title_full DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
title_fullStr DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
title_full_unstemmed DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
title_short DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
title_sort drcm: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
topic Physiology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10656763/
https://www.ncbi.nlm.nih.gov/pubmed/38028809
http://dx.doi.org/10.3389/fphys.2023.1241370
work_keys_str_mv AT huangwanwan drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion
AT zhanghan drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion
AT chengyu drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion
AT quanxiongwen drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion