Cargando…
A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs
Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studie...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
MDPI
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8954226/ https://www.ncbi.nlm.nih.gov/pubmed/35336270 http://dx.doi.org/10.3390/s22062099 |
_version_ | 1784676042567319552 |
---|---|
author | Fan, Yunsheng Sun, Zhe Wang, Guofeng |
author_facet | Fan, Yunsheng Sun, Zhe Wang, Guofeng |
author_sort | Fan, Yunsheng |
collection | PubMed |
description | Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs’ maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal. |
format | Online Article Text |
id | pubmed-8954226 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | MDPI |
record_format | MEDLINE/PubMed |
spelling | pubmed-89542262022-03-26 A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs Fan, Yunsheng Sun, Zhe Wang, Guofeng Sensors (Basel) Article Autonomous collision avoidance technology provides an intelligent method for unmanned surface vehicles’ (USVs) safe and efficient navigation. In this paper, the USV collision avoidance problem under the constraint of the international regulations for preventing collisions at sea (COLREGs) was studied. Here, a reinforcement learning collision avoidance (RLCA) algorithm is proposed that complies with USV maneuverability. Notably, the reinforcement learning agent does not require any prior knowledge about USV collision avoidance from humans to learn collision avoidance motions well. The double-DQN method was used to reduce the overestimation of the action-value function. A dueling network architecture was adopted to clearly distinguish the difference between a great state and an excellent action. Aiming at the problem of agent exploration, a method based on the characteristics of USV collision avoidance, the category-based exploration method, can improve the exploration ability of the USV. Because a large number of turning behaviors in the early steps may affect the training, a method to discard some of the transitions was designed, which can improve the effectiveness of the algorithm. A finite Markov decision process (MDP) that conforms to the USVs’ maneuverability and COLREGs was used for the agent training. The RLCA algorithm was tested in a marine simulation environment in many different USV encounters, which showed a higher average reward. The RLCA algorithm bridged the divide between USV navigation status information and collision avoidance behavior, resulting in successfully planning a safe and economical path to the terminal. MDPI 2022-03-08 /pmc/articles/PMC8954226/ /pubmed/35336270 http://dx.doi.org/10.3390/s22062099 Text en © 2022 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Article Fan, Yunsheng Sun, Zhe Wang, Guofeng A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs |
title | A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs |
title_full | A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs |
title_fullStr | A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs |
title_full_unstemmed | A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs |
title_short | A Novel Reinforcement Learning Collision Avoidance Algorithm for USVs Based on Maneuvering Characteristics and COLREGs |
title_sort | novel reinforcement learning collision avoidance algorithm for usvs based on maneuvering characteristics and colregs |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8954226/ https://www.ncbi.nlm.nih.gov/pubmed/35336270 http://dx.doi.org/10.3390/s22062099 |
work_keys_str_mv | AT fanyunsheng anovelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs AT sunzhe anovelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs AT wangguofeng anovelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs AT fanyunsheng novelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs AT sunzhe novelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs AT wangguofeng novelreinforcementlearningcollisionavoidancealgorithmforusvsbasedonmaneuveringcharacteristicsandcolregs |