Cargando…

Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks

Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are r...

Descripción completa

Detalles Bibliográficos
Autores principales: Yoder, Jason A., Anderson, Cooper B., Wang, Cehong, Izquierdo, Eduardo J.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9028035/
https://www.ncbi.nlm.nih.gov/pubmed/35465269
http://dx.doi.org/10.3389/fncom.2022.818985
_version_ 1784691516610969600
author Yoder, Jason A.
Anderson, Cooper B.
Wang, Cehong
Izquierdo, Eduardo J.
author_facet Yoder, Jason A.
Anderson, Cooper B.
Wang, Cehong
Izquierdo, Eduardo J.
author_sort Yoder, Jason A.
collection PubMed
description Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are recurrent, dynamic, and non-linear and models of neural circuits employed in neuroscience and neuroethology tend to involve, accordingly, continuous-time, non-linear, and recurrently interconnected components. Currently, the main approach for finding configurations of dynamical recurrent neural networks that demonstrate behaviors of interest is using stochastic search techniques, such as evolutionary algorithms. In an evolutionary algorithm, these dynamic recurrent neural networks are evolved to perform the behavior over multiple generations, through selection, inheritance, and mutation, across a population of solutions. Although, these systems can be evolved to exhibit lifetime learning behavior, there are no explicit rules built into these dynamic recurrent neural networks that facilitate learning during their lifetime (e.g., reward signals). In this work, we examine a biologically plausible lifetime learning mechanism for dynamical recurrent neural networks. We focus on a recently proposed reinforcement learning mechanism inspired by neuromodulatory reward signals and ongoing fluctuations in synaptic strengths. Specifically, we extend one of the best-studied and most-commonly used dynamic recurrent neural networks to incorporate the reinforcement learning mechanism. First, we demonstrate that this extended dynamical system (model and learning mechanism) can autonomously learn to perform a central pattern generation task. Second, we compare the robustness and efficiency of the reinforcement learning rules in relation to two baseline models, a random walk and a hill-climbing walk through parameter space. Third, we systematically study the effect of the different meta-parameters of the learning mechanism on the behavioral learning performance. Finally, we report on preliminary results exploring the generality and scalability of this learning mechanism for dynamical neural networks as well as directions for future work.
format Online
Article
Text
id pubmed-9028035
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-90280352022-04-23 Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks Yoder, Jason A. Anderson, Cooper B. Wang, Cehong Izquierdo, Eduardo J. Front Comput Neurosci Neuroscience Lifetime learning, or the change (or acquisition) of behaviors during a lifetime, based on experience, is a hallmark of living organisms. Multiple mechanisms may be involved, but biological neural circuits have repeatedly demonstrated a vital role in the learning process. These neural circuits are recurrent, dynamic, and non-linear and models of neural circuits employed in neuroscience and neuroethology tend to involve, accordingly, continuous-time, non-linear, and recurrently interconnected components. Currently, the main approach for finding configurations of dynamical recurrent neural networks that demonstrate behaviors of interest is using stochastic search techniques, such as evolutionary algorithms. In an evolutionary algorithm, these dynamic recurrent neural networks are evolved to perform the behavior over multiple generations, through selection, inheritance, and mutation, across a population of solutions. Although, these systems can be evolved to exhibit lifetime learning behavior, there are no explicit rules built into these dynamic recurrent neural networks that facilitate learning during their lifetime (e.g., reward signals). In this work, we examine a biologically plausible lifetime learning mechanism for dynamical recurrent neural networks. We focus on a recently proposed reinforcement learning mechanism inspired by neuromodulatory reward signals and ongoing fluctuations in synaptic strengths. Specifically, we extend one of the best-studied and most-commonly used dynamic recurrent neural networks to incorporate the reinforcement learning mechanism. First, we demonstrate that this extended dynamical system (model and learning mechanism) can autonomously learn to perform a central pattern generation task. Second, we compare the robustness and efficiency of the reinforcement learning rules in relation to two baseline models, a random walk and a hill-climbing walk through parameter space. Third, we systematically study the effect of the different meta-parameters of the learning mechanism on the behavioral learning performance. Finally, we report on preliminary results exploring the generality and scalability of this learning mechanism for dynamical neural networks as well as directions for future work. Frontiers Media S.A. 2022-04-08 /pmc/articles/PMC9028035/ /pubmed/35465269 http://dx.doi.org/10.3389/fncom.2022.818985 Text en Copyright © 2022 Yoder, Anderson, Wang and Izquierdo. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Neuroscience
Yoder, Jason A.
Anderson, Cooper B.
Wang, Cehong
Izquierdo, Eduardo J.
Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks
title Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks
title_full Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks
title_fullStr Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks
title_full_unstemmed Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks
title_short Reinforcement Learning for Central Pattern Generation in Dynamical Recurrent Neural Networks
title_sort reinforcement learning for central pattern generation in dynamical recurrent neural networks
topic Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9028035/
https://www.ncbi.nlm.nih.gov/pubmed/35465269
http://dx.doi.org/10.3389/fncom.2022.818985
work_keys_str_mv AT yoderjasona reinforcementlearningforcentralpatterngenerationindynamicalrecurrentneuralnetworks
AT andersoncooperb reinforcementlearningforcentralpatterngenerationindynamicalrecurrentneuralnetworks
AT wangcehong reinforcementlearningforcentralpatterngenerationindynamicalrecurrentneuralnetworks
AT izquierdoeduardoj reinforcementlearningforcentralpatterngenerationindynamicalrecurrentneuralnetworks