Cargando…

A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning

Humans do not learn everything from the scratch but can connect and associate the upcoming information with the exchanged experience and known knowledge. Such an idea can be extended to cooperated multi-reinforcement learning and has achieved its success on homogeneous agents by means of parameter s...

Descripción completa

Detalles Bibliográficos
Autores principales: Zhao, Biao, Jin, Weiqiang, Chen, Zhang, Guo, Yucheng
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10315621/
https://www.ncbi.nlm.nih.gov/pubmed/37404464
http://dx.doi.org/10.3389/fnins.2023.1201370
_version_ 1785067536175333376
author Zhao, Biao
Jin, Weiqiang
Chen, Zhang
Guo, Yucheng
author_facet Zhao, Biao
Jin, Weiqiang
Chen, Zhang
Guo, Yucheng
author_sort Zhao, Biao
collection PubMed
description Humans do not learn everything from the scratch but can connect and associate the upcoming information with the exchanged experience and known knowledge. Such an idea can be extended to cooperated multi-reinforcement learning and has achieved its success on homogeneous agents by means of parameter sharing. However, it is difficult to straightforwardly apply parameter sharing when dealing with heterogeneous agents thanks to their individual forms of input/output and their diverse functions and targets. Neuroscience has provided evidence that our brain creates several levels of experience and knowledge-sharing mechanisms that not only exchange similar experiences but also allow for sharing of abstract concepts to handle unfamiliar situations that others have already encountered. Inspired by such a brain's functions, we propose a semi-independent training policy method that can well tackle the conflict between parameter sharing and specialized training for heterogeneous agents. It employs a shared common representation for both observation and action, enabling the integration of various input and output sources. Additionally, a shared latent space is utilized to maintain a balanced relationship between the upstream policy and downstream functions, benefiting each individual agent's target. From the experiments, it can approve that our proposed method outperforms the current mainstream algorithms, especially when handling heterogeneous agents. Empirically, our proposed method can also be improved as a more general and fundamental heterogeneous agents' reinforcement learning structure for curriculum learning and representation transfer. All our code is open and released on https://gitlab.com/reinforcement/ntype.
format Online
Article
Text
id pubmed-10315621
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-103156212023-07-04 A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning Zhao, Biao Jin, Weiqiang Chen, Zhang Guo, Yucheng Front Neurosci Neuroscience Humans do not learn everything from the scratch but can connect and associate the upcoming information with the exchanged experience and known knowledge. Such an idea can be extended to cooperated multi-reinforcement learning and has achieved its success on homogeneous agents by means of parameter sharing. However, it is difficult to straightforwardly apply parameter sharing when dealing with heterogeneous agents thanks to their individual forms of input/output and their diverse functions and targets. Neuroscience has provided evidence that our brain creates several levels of experience and knowledge-sharing mechanisms that not only exchange similar experiences but also allow for sharing of abstract concepts to handle unfamiliar situations that others have already encountered. Inspired by such a brain's functions, we propose a semi-independent training policy method that can well tackle the conflict between parameter sharing and specialized training for heterogeneous agents. It employs a shared common representation for both observation and action, enabling the integration of various input and output sources. Additionally, a shared latent space is utilized to maintain a balanced relationship between the upstream policy and downstream functions, benefiting each individual agent's target. From the experiments, it can approve that our proposed method outperforms the current mainstream algorithms, especially when handling heterogeneous agents. Empirically, our proposed method can also be improved as a more general and fundamental heterogeneous agents' reinforcement learning structure for curriculum learning and representation transfer. All our code is open and released on https://gitlab.com/reinforcement/ntype. Frontiers Media S.A. 2023-06-19 /pmc/articles/PMC10315621/ /pubmed/37404464 http://dx.doi.org/10.3389/fnins.2023.1201370 Text en Copyright © 2023 Zhao, Jin, Chen and Guo. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Neuroscience
Zhao, Biao
Jin, Weiqiang
Chen, Zhang
Guo, Yucheng
A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
title A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
title_full A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
title_fullStr A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
title_full_unstemmed A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
title_short A semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
title_sort semi-independent policies training method with shared representation for heterogeneous multi-agents reinforcement learning
topic Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10315621/
https://www.ncbi.nlm.nih.gov/pubmed/37404464
http://dx.doi.org/10.3389/fnins.2023.1201370
work_keys_str_mv AT zhaobiao asemiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT jinweiqiang asemiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT chenzhang asemiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT guoyucheng asemiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT zhaobiao semiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT jinweiqiang semiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT chenzhang semiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning
AT guoyucheng semiindependentpoliciestrainingmethodwithsharedrepresentationforheterogeneousmultiagentsreinforcementlearning