Cargando…

Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion

When traditional super-resolution reconstruction methods are applied to infrared thermal images, they often ignore the problem of poor image quality caused by the imaging mechanism, which makes it difficult to obtain high-quality reconstruction results even with the training of simulated degraded in...

Descripción completa

Detalles Bibliográficos
Autores principales: Jiang, Yichun, Liu, Yunqing, Zhan, Weida, Zhu, Depeng
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10297407/
https://www.ncbi.nlm.nih.gov/pubmed/37372258
http://dx.doi.org/10.3390/e25060914
_version_ 1785063876431183872
author Jiang, Yichun
Liu, Yunqing
Zhan, Weida
Zhu, Depeng
author_facet Jiang, Yichun
Liu, Yunqing
Zhan, Weida
Zhu, Depeng
author_sort Jiang, Yichun
collection PubMed
description When traditional super-resolution reconstruction methods are applied to infrared thermal images, they often ignore the problem of poor image quality caused by the imaging mechanism, which makes it difficult to obtain high-quality reconstruction results even with the training of simulated degraded inverse processes. To address these issues, we proposed a thermal infrared image super-resolution reconstruction method based on multimodal sensor fusion, aiming to enhance the resolution of thermal infrared images and rely on multimodal sensor information to reconstruct high-frequency details in the images, thereby overcoming the limitations of imaging mechanisms. First, we designed a novel super-resolution reconstruction network, which consisted of primary feature encoding, super-resolution reconstruction, and high-frequency detail fusion subnetwork, to enhance the resolution of thermal infrared images and rely on multimodal sensor information to reconstruct high-frequency details in the images, thereby overcoming limitations of imaging mechanisms. We designed hierarchical dilated distillation modules and a cross-attention transformation module to extract and transmit image features, enhancing the network’s ability to express complex patterns. Then, we proposed a hybrid loss function to guide the network in extracting salient features from thermal infrared images and reference images while maintaining accurate thermal information. Finally, we proposed a learning strategy to ensure the high-quality super-resolution reconstruction performance of the network, even in the absence of reference images. Extensive experimental results show that the proposed method exhibits superior reconstruction image quality compared to other contrastive methods, demonstrating its effectiveness.
format Online
Article
Text
id pubmed-10297407
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-102974072023-06-28 Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion Jiang, Yichun Liu, Yunqing Zhan, Weida Zhu, Depeng Entropy (Basel) Article When traditional super-resolution reconstruction methods are applied to infrared thermal images, they often ignore the problem of poor image quality caused by the imaging mechanism, which makes it difficult to obtain high-quality reconstruction results even with the training of simulated degraded inverse processes. To address these issues, we proposed a thermal infrared image super-resolution reconstruction method based on multimodal sensor fusion, aiming to enhance the resolution of thermal infrared images and rely on multimodal sensor information to reconstruct high-frequency details in the images, thereby overcoming the limitations of imaging mechanisms. First, we designed a novel super-resolution reconstruction network, which consisted of primary feature encoding, super-resolution reconstruction, and high-frequency detail fusion subnetwork, to enhance the resolution of thermal infrared images and rely on multimodal sensor information to reconstruct high-frequency details in the images, thereby overcoming limitations of imaging mechanisms. We designed hierarchical dilated distillation modules and a cross-attention transformation module to extract and transmit image features, enhancing the network’s ability to express complex patterns. Then, we proposed a hybrid loss function to guide the network in extracting salient features from thermal infrared images and reference images while maintaining accurate thermal information. Finally, we proposed a learning strategy to ensure the high-quality super-resolution reconstruction performance of the network, even in the absence of reference images. Extensive experimental results show that the proposed method exhibits superior reconstruction image quality compared to other contrastive methods, demonstrating its effectiveness. MDPI 2023-06-09 /pmc/articles/PMC10297407/ /pubmed/37372258 http://dx.doi.org/10.3390/e25060914 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Jiang, Yichun
Liu, Yunqing
Zhan, Weida
Zhu, Depeng
Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion
title Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion
title_full Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion
title_fullStr Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion
title_full_unstemmed Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion
title_short Improved Thermal Infrared Image Super-Resolution Reconstruction Method Base on Multimodal Sensor Fusion
title_sort improved thermal infrared image super-resolution reconstruction method base on multimodal sensor fusion
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10297407/
https://www.ncbi.nlm.nih.gov/pubmed/37372258
http://dx.doi.org/10.3390/e25060914
work_keys_str_mv AT jiangyichun improvedthermalinfraredimagesuperresolutionreconstructionmethodbaseonmultimodalsensorfusion
AT liuyunqing improvedthermalinfraredimagesuperresolutionreconstructionmethodbaseonmultimodalsensorfusion
AT zhanweida improvedthermalinfraredimagesuperresolutionreconstructionmethodbaseonmultimodalsensorfusion
AT zhudepeng improvedthermalinfraredimagesuperresolutionreconstructionmethodbaseonmultimodalsensorfusion