Cargando…
DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion
Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special lear...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10656763/ https://www.ncbi.nlm.nih.gov/pubmed/38028809 http://dx.doi.org/10.3389/fphys.2023.1241370 |
_version_ | 1785137070610579456 |
---|---|
author | Huang, Wanwan Zhang, Han Cheng, Yu Quan, Xiongwen |
author_facet | Huang, Wanwan Zhang, Han Cheng, Yu Quan, Xiongwen |
author_sort | Huang, Wanwan |
collection | PubMed |
description | Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism-based disentangled representation network for medical image fusion, which designs coordinate attention and multimodal attention to extract and strengthen common and exclusive features. First, the common and exclusive features of each modality were obtained by the cross mutual information and adversarial objective methods, respectively. Then, coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, these two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis, and four metrics are used for quantitative comparison. The values of the four metrics demonstrate the effect of the DRCM. Furthermore, the DRCM achieved better results on SCD, Nabf, and MS-SSIM metrics, which indicates that the DRCM achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, it was found that the DRCM outperforms the comparison method. |
format | Online Article Text |
id | pubmed-10656763 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-106567632023-11-03 DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion Huang, Wanwan Zhang, Han Cheng, Yu Quan, Xiongwen Front Physiol Physiology Recent studies on medical image fusion based on deep learning have made remarkable progress, but the common and exclusive features of different modalities, especially their subsequent feature enhancement, are ignored. Since medical images of different modalities have unique information, special learning of exclusive features should be designed to express the unique information of different modalities so as to obtain a medical fusion image with more information and details. Therefore, we propose an attention mechanism-based disentangled representation network for medical image fusion, which designs coordinate attention and multimodal attention to extract and strengthen common and exclusive features. First, the common and exclusive features of each modality were obtained by the cross mutual information and adversarial objective methods, respectively. Then, coordinate attention is focused on the enhancement of the common and exclusive features of different modalities, and the exclusive features are weighted by multimodal attention. Finally, these two kinds of features are fused. The effectiveness of the three innovation modules is verified by ablation experiments. Furthermore, eight comparison methods are selected for qualitative analysis, and four metrics are used for quantitative comparison. The values of the four metrics demonstrate the effect of the DRCM. Furthermore, the DRCM achieved better results on SCD, Nabf, and MS-SSIM metrics, which indicates that the DRCM achieved the goal of further improving the visual quality of the fused image with more information from source images and less noise. Through the comprehensive comparison and analysis of the experimental results, it was found that the DRCM outperforms the comparison method. Frontiers Media S.A. 2023-11-03 /pmc/articles/PMC10656763/ /pubmed/38028809 http://dx.doi.org/10.3389/fphys.2023.1241370 Text en Copyright © 2023 Huang, Zhang, Cheng and Quan. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Physiology Huang, Wanwan Zhang, Han Cheng, Yu Quan, Xiongwen DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
title | DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
title_full | DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
title_fullStr | DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
title_full_unstemmed | DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
title_short | DRCM: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
title_sort | drcm: a disentangled representation network based on coordinate and multimodal attention for medical image fusion |
topic | Physiology |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10656763/ https://www.ncbi.nlm.nih.gov/pubmed/38028809 http://dx.doi.org/10.3389/fphys.2023.1241370 |
work_keys_str_mv | AT huangwanwan drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion AT zhanghan drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion AT chengyu drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion AT quanxiongwen drcmadisentangledrepresentationnetworkbasedoncoordinateandmultimodalattentionformedicalimagefusion |