Cargando…
An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles
The marine environment presents a unique set of challenges for human–robot interaction. Communicating with gestures is a common way for interacting between the diver and autonomous underwater vehicles (AUVs). However, underwater gesture recognition is a challenging visual task for AUVs due to light...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9824386/ https://www.ncbi.nlm.nih.gov/pubmed/36616794 http://dx.doi.org/10.3390/s23010197 |
_version_ | 1784866397107519488 |
---|---|
author | Zhang, Yongji Jiang, Yu Qi, Hong Zhao, Minghao Wang, Yuehang Wang, Kai Wei, Fenglin |
author_facet | Zhang, Yongji Jiang, Yu Qi, Hong Zhao, Minghao Wang, Yuehang Wang, Kai Wei, Fenglin |
author_sort | Zhang, Yongji |
collection | PubMed |
description | The marine environment presents a unique set of challenges for human–robot interaction. Communicating with gestures is a common way for interacting between the diver and autonomous underwater vehicles (AUVs). However, underwater gesture recognition is a challenging visual task for AUVs due to light refraction and wavelength color attenuation issues. Current gesture recognition methods classify the whole image directly or locate the hand position first and then classify the hand features. Among these purely visual approaches, textual information is largely ignored. This paper proposes a visual–textual model for underwater hand gesture recognition (VT-UHGR). The VT-UHGR model encodes the underwater diver’s image as visual features, the category text as textual features, and generates visual–textual features through multimodal interactions. We guide AUVs to use image–text matching for learning and inference. The proposed method achieves better performance than most existing purely visual methods on the dataset CADDY, demonstrating the effectiveness of using textual patterns for underwater gesture recognition. |
format | Online Article Text |
id | pubmed-9824386 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-98243862023-01-08 An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles Zhang, Yongji Jiang, Yu Qi, Hong Zhao, Minghao Wang, Yuehang Wang, Kai Wei, Fenglin Sensors (Basel) Article The marine environment presents a unique set of challenges for human–robot interaction. Communicating with gestures is a common way for interacting between the diver and autonomous underwater vehicles (AUVs). However, underwater gesture recognition is a challenging visual task for AUVs due to light refraction and wavelength color attenuation issues. Current gesture recognition methods classify the whole image directly or locate the hand position first and then classify the hand features. Among these purely visual approaches, textual information is largely ignored. This paper proposes a visual–textual model for underwater hand gesture recognition (VT-UHGR). The VT-UHGR model encodes the underwater diver’s image as visual features, the category text as textual features, and generates visual–textual features through multimodal interactions. We guide AUVs to use image–text matching for learning and inference. The proposed method achieves better performance than most existing purely visual methods on the dataset CADDY, demonstrating the effectiveness of using textual patterns for underwater gesture recognition. MDPI 2022-12-24 /pmc/articles/PMC9824386/ /pubmed/36616794 http://dx.doi.org/10.3390/s23010197 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Zhang, Yongji Jiang, Yu Qi, Hong Zhao, Minghao Wang, Yuehang Wang, Kai Wei, Fenglin An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles |
title | An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles |
title_full | An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles |
title_fullStr | An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles |
title_full_unstemmed | An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles |
title_short | An Underwater Human–Robot Interaction Using a Visual–Textual Model for Autonomous Underwater Vehicles |
title_sort | underwater human–robot interaction using a visual–textual model for autonomous underwater vehicles |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9824386/ https://www.ncbi.nlm.nih.gov/pubmed/36616794 http://dx.doi.org/10.3390/s23010197 |
work_keys_str_mv | AT zhangyongji anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT jiangyu anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT qihong anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT zhaominghao anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT wangyuehang anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT wangkai anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT weifenglin anunderwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT zhangyongji underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT jiangyu underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT qihong underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT zhaominghao underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT wangyuehang underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT wangkai underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles AT weifenglin underwaterhumanrobotinteractionusingavisualtextualmodelforautonomousunderwatervehicles |