Cargando…
Infrared Image Caption Based on Object-Oriented Attention
With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practic...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10216905/ https://www.ncbi.nlm.nih.gov/pubmed/37238581 http://dx.doi.org/10.3390/e25050826 |
_version_ | 1785048408068718592 |
---|---|
author | Lv, Junfeng Hui, Tian Zhi, Yongfeng Xu, Yuelei |
author_facet | Lv, Junfeng Hui, Tian Zhi, Yongfeng Xu, Yuelei |
author_sort | Lv, Junfeng |
collection | PubMed |
description | With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practical task is widely used in night security, as well as for understanding night scenes and other scenarios. However, due to the differences in image features and the complexity of semantic information, generating captions for infrared images remains a challenging task. From the perspective of deployment and application, to improve the correlation between descriptions and objects, we introduced the YOLOv6 and LSTM as encoder-decoder structure and proposed infrared image caption based on object-oriented attention. Firstly, to improve the domain adaptability of the detector, we optimized the pseudo-label learning process. Secondly, we proposed the object-oriented attention method to address the alignment problem between complex semantic information and embedded words. This method helps select the most crucial features of the object region and guides the caption model in generating words that are more relevant to the object. Our methods have shown good performance on the infrared image and can produce words explicitly associated with the object regions located by the detector. The robustness and effectiveness of the proposed methods were demonstrated through evaluation on various datasets, along with other state-of-the-art methods. Our approach achieved BLUE-4 scores of 31.6 and 41.2 on KAIST and Infrared City and Town datasets, respectively. Our approach provides a feasible solution for the deployment of embedded devices in industrial applications. |
format | Online Article Text |
id | pubmed-10216905 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-102169052023-05-27 Infrared Image Caption Based on Object-Oriented Attention Lv, Junfeng Hui, Tian Zhi, Yongfeng Xu, Yuelei Entropy (Basel) Article With the ongoing development of image technology, the deployment of various intelligent applications on embedded devices has attracted increased attention in the industry. One such application is automatic image captioning for infrared images, which involves converting images into text. This practical task is widely used in night security, as well as for understanding night scenes and other scenarios. However, due to the differences in image features and the complexity of semantic information, generating captions for infrared images remains a challenging task. From the perspective of deployment and application, to improve the correlation between descriptions and objects, we introduced the YOLOv6 and LSTM as encoder-decoder structure and proposed infrared image caption based on object-oriented attention. Firstly, to improve the domain adaptability of the detector, we optimized the pseudo-label learning process. Secondly, we proposed the object-oriented attention method to address the alignment problem between complex semantic information and embedded words. This method helps select the most crucial features of the object region and guides the caption model in generating words that are more relevant to the object. Our methods have shown good performance on the infrared image and can produce words explicitly associated with the object regions located by the detector. The robustness and effectiveness of the proposed methods were demonstrated through evaluation on various datasets, along with other state-of-the-art methods. Our approach achieved BLUE-4 scores of 31.6 and 41.2 on KAIST and Infrared City and Town datasets, respectively. Our approach provides a feasible solution for the deployment of embedded devices in industrial applications. MDPI 2023-05-22 /pmc/articles/PMC10216905/ /pubmed/37238581 http://dx.doi.org/10.3390/e25050826 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Lv, Junfeng Hui, Tian Zhi, Yongfeng Xu, Yuelei Infrared Image Caption Based on Object-Oriented Attention |
title | Infrared Image Caption Based on Object-Oriented Attention |
title_full | Infrared Image Caption Based on Object-Oriented Attention |
title_fullStr | Infrared Image Caption Based on Object-Oriented Attention |
title_full_unstemmed | Infrared Image Caption Based on Object-Oriented Attention |
title_short | Infrared Image Caption Based on Object-Oriented Attention |
title_sort | infrared image caption based on object-oriented attention |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10216905/ https://www.ncbi.nlm.nih.gov/pubmed/37238581 http://dx.doi.org/10.3390/e25050826 |
work_keys_str_mv | AT lvjunfeng infraredimagecaptionbasedonobjectorientedattention AT huitian infraredimagecaptionbasedonobjectorientedattention AT zhiyongfeng infraredimagecaptionbasedonobjectorientedattention AT xuyuelei infraredimagecaptionbasedonobjectorientedattention |