Cargando…

Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization

To solve the problems of path planning and dynamic obstacle avoidance for an unmanned surface vehicle (USV) in a locally observable non-dynamic ocean environment, a visual perception and decision-making method based on deep reinforcement learning is proposed. This method replaces the full connection...

Descripción completa

Detalles Bibliográficos
Autores principales: Sun, Pengqi, Yang, Chunxi, Zhou, Xiaojie, Wang, Wenbo
Formato: Online Artículo Texto
Lenguaje:English
Publicado: MDPI 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10650858/
https://www.ncbi.nlm.nih.gov/pubmed/37960565
http://dx.doi.org/10.3390/s23218864
_version_ 1785135877928779776
author Sun, Pengqi
Yang, Chunxi
Zhou, Xiaojie
Wang, Wenbo
author_facet Sun, Pengqi
Yang, Chunxi
Zhou, Xiaojie
Wang, Wenbo
author_sort Sun, Pengqi
collection PubMed
description To solve the problems of path planning and dynamic obstacle avoidance for an unmanned surface vehicle (USV) in a locally observable non-dynamic ocean environment, a visual perception and decision-making method based on deep reinforcement learning is proposed. This method replaces the full connection layer in the Proximal Policy Optimization (PPO) neural network structure with a convolutional neural network (CNN). In this way, the degree of memorization and forgetting of sample information is controlled. Moreover, this method accumulates reward models faster by preferentially learning samples with high reward values. From the USV-centered radar perception input of the local environment, the output of the action is realized through an end-to-end learning model, and the environment perception and decision are formed as a closed loop. Thus, the proposed algorithm has good adaptability in different marine environments. The simulation results show that, compared with the PPO algorithm, Soft Actor–Critic (SAC) algorithm, and Deep Q Network (DQN) algorithm, the proposed algorithm can accelerate the model convergence speed and improve the path planning performances in partly or fully unknown ocean fields.
format Online
Article
Text
id pubmed-10650858
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher MDPI
record_format MEDLINE/PubMed
spelling pubmed-106508582023-10-31 Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization Sun, Pengqi Yang, Chunxi Zhou, Xiaojie Wang, Wenbo Sensors (Basel) Article To solve the problems of path planning and dynamic obstacle avoidance for an unmanned surface vehicle (USV) in a locally observable non-dynamic ocean environment, a visual perception and decision-making method based on deep reinforcement learning is proposed. This method replaces the full connection layer in the Proximal Policy Optimization (PPO) neural network structure with a convolutional neural network (CNN). In this way, the degree of memorization and forgetting of sample information is controlled. Moreover, this method accumulates reward models faster by preferentially learning samples with high reward values. From the USV-centered radar perception input of the local environment, the output of the action is realized through an end-to-end learning model, and the environment perception and decision are formed as a closed loop. Thus, the proposed algorithm has good adaptability in different marine environments. The simulation results show that, compared with the PPO algorithm, Soft Actor–Critic (SAC) algorithm, and Deep Q Network (DQN) algorithm, the proposed algorithm can accelerate the model convergence speed and improve the path planning performances in partly or fully unknown ocean fields. MDPI 2023-10-31 /pmc/articles/PMC10650858/ /pubmed/37960565 http://dx.doi.org/10.3390/s23218864 Text en © 2023 by the authors. https://creativecommons.org/licenses/by/4.0/Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Sun, Pengqi
Yang, Chunxi
Zhou, Xiaojie
Wang, Wenbo
Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization
title Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization
title_full Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization
title_fullStr Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization
title_full_unstemmed Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization
title_short Path Planning for Unmanned Surface Vehicles with Strong Generalization Ability Based on Improved Proximal Policy Optimization
title_sort path planning for unmanned surface vehicles with strong generalization ability based on improved proximal policy optimization
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10650858/
https://www.ncbi.nlm.nih.gov/pubmed/37960565
http://dx.doi.org/10.3390/s23218864
work_keys_str_mv AT sunpengqi pathplanningforunmannedsurfacevehicleswithstronggeneralizationabilitybasedonimprovedproximalpolicyoptimization
AT yangchunxi pathplanningforunmannedsurfacevehicleswithstronggeneralizationabilitybasedonimprovedproximalpolicyoptimization
AT zhouxiaojie pathplanningforunmannedsurfacevehicleswithstronggeneralizationabilitybasedonimprovedproximalpolicyoptimization
AT wangwenbo pathplanningforunmannedsurfacevehicleswithstronggeneralizationabilitybasedonimprovedproximalpolicyoptimization