Cargando…

A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs

Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studie...

Descripción completa

Detalles Bibliográficos
Autores principales: Fan, Yunsheng, Sun, Zhe, Wang, Guofeng
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8954226/
https://www.ncbi.nlm.nih.gov/pubmed/35336270
http://dx.doi.org/10.3390/s22062099
_version_ 1784676042567319552
author Fan, Yunsheng
Sun, Zhe
Wang, Guofeng
author_facet Fan, Yunsheng
Sun, Zhe
Wang, Guofeng
author_sort Fan, Yunsheng
collection PubMed
description Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs’ maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal.
format Online
Article
Text
id pubmed-8954226
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-89542262022-03-26 A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs Fan, Yunsheng Sun, Zhe Wang, Guofeng Sensors (Basel) Article Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs’ maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal. MDPI 2022-03-08 /pmc/articles/PMC8954226/ /pubmed/35336270 http://dx.doi.org/10.3390/s22062099 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Fan, Yunsheng
Sun, Zhe
Wang, Guofeng
A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
title A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
title_full A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
title_fullStr A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
title_full_unstemmed A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
title_short A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
title_sort novel reinforcement learning collision avoidance algorithm for usvs based on maneuvering characteristics and colregs
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8954226/
https://www.ncbi.nlm.nih.gov/pubmed/35336270
http://dx.doi.org/10.3390/s22062099
work_keys_str_mv AT fanyunsheng anovelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs
AT sunzhe anovelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs
AT wangguofeng anovelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs
AT fanyunsheng novelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs
AT sunzhe novelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs
AT wangguofeng novelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs