Cargando…

Infrared Image Caption Based on Object-Oriented Attention

With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practic...

Descripción completa

Detalles Bibliográficos
Autores principales: Lv, Junfeng, Hui, Tian, Zhi, Yongfeng, Xu, Yuelei
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10216905/
https://www.ncbi.nlm.nih.gov/pubmed/37238581
http://dx.doi.org/10.3390/e25050826
_version_ 1785048408068718592
author Lv, Junfeng
Hui, Tian
Zhi, Yongfeng
Xu, Yuelei
author_facet Lv, Junfeng
Hui, Tian
Zhi, Yongfeng
Xu, Yuelei
author_sort Lv, Junfeng
collection PubMed
description With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practical task is widely used in night security, as well as for understanding night scenes and other scenarios. However, due to the differences in image features and the complexity of semantic information, generating captions for infrared images remains a challenging task. From the perspective of deployment and application, to improve the correlation between descriptions and objects, we introduced the YOLOv6 and LSTM as encoder-decoder structure and proposed infrared image caption based on object-oriented attention. Firstly, to improve the domain adaptability of the detector, we optimized the pseudo-label learning process. Secondly, we proposed the object-oriented attention method to address the alignment problem between complex semantic information and embedded words. This method helps select the most crucial features of the object region and guides the caption model in generating words that are more relevant to the object. Our methods have shown good performance on the infrared image and can produce words explicitly associated with the object regions located by the detector. The robustness and effectiveness of the proposed methods were demonstrated through evaluation on various datasets, along with other state-of-the-art methods. Our approach achieved BLUE-4 scores of 31.6 and 41.2 on KAIST and Infrared City and Town datasets, respectively. Our approach provides a feasible solution for the deployment of embedded devices in industrial applications.
format Online
Article
Text
id pubmed-10216905
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-102169052023-05-27 Infrared Image Caption Based on Object-Oriented Attention Lv, Junfeng Hui, Tian Zhi, Yongfeng Xu, Yuelei Entropy (Basel) Article With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practical task is widely used in night security, as well as for understanding night scenes and other scenarios. However, due to the differences in image features and the complexity of semantic information, generating captions for infrared images remains a challenging task. From the perspective of deployment and application, to improve the correlation between descriptions and objects, we introduced the YOLOv6 and LSTM as encoder-decoder structure and proposed infrared image caption based on object-oriented attention. Firstly, to improve the domain adaptability of the detector, we optimized the pseudo-label learning process. Secondly, we proposed the object-oriented attention method to address the alignment problem between complex semantic information and embedded words. This method helps select the most crucial features of the object region and guides the caption model in generating words that are more relevant to the object. Our methods have shown good performance on the infrared image and can produce words explicitly associated with the object regions located by the detector. The robustness and effectiveness of the proposed methods were demonstrated through evaluation on various datasets, along with other state-of-the-art methods. Our approach achieved BLUE-4 scores of 31.6 and 41.2 on KAIST and Infrared City and Town datasets, respectively. Our approach provides a feasible solution for the deployment of embedded devices in industrial applications. MDPI 2023-05-22 /pmc/articles/PMC10216905/ /pubmed/37238581 http://dx.doi.org/10.3390/e25050826 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Lv, Junfeng
Hui, Tian
Zhi, Yongfeng
Xu, Yuelei
Infrared Image Caption Based on Object-Oriented Attention
title Infrared Image Caption Based on Object-Oriented Attention
title_full Infrared Image Caption Based on Object-Oriented Attention
title_fullStr Infrared Image Caption Based on Object-Oriented Attention
title_full_unstemmed Infrared Image Caption Based on Object-Oriented Attention
title_short Infrared Image Caption Based on Object-Oriented Attention
title_sort infrared image caption based on object-oriented attention
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10216905/
https://www.ncbi.nlm.nih.gov/pubmed/37238581
http://dx.doi.org/10.3390/e25050826
work_keys_str_mv AT lvjunfeng infraredimagecaptionbasedonobjectorientedattention
AT huitian infraredimagecaptionbasedonobjectorientedattention
AT zhiyongfeng infraredimagecaptionbasedonobjectorientedattention
AT xuyuelei infraredimagecaptionbasedonobjectorientedattention