Cargando…
Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Public Library of Science
2012
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477154/ https://www.ncbi.nlm.nih.gov/pubmed/23094042 http://dx.doi.org/10.1371/journal.pone.0047251 |
_version_ | 1782247205067816960 |
---|---|
author | Chadderdon, George L. Neymotin, Samuel A. Kerr, Cliff C. Lytton, William W. |
author_facet | Chadderdon, George L. Neymotin, Samuel A. Kerr, Cliff C. Lytton, William W. |
author_sort | Chadderdon, George L. |
collection | PubMed |
description | Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (−1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior. |
format | Online Article Text |
id | pubmed-3477154 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2012 |
publisher | Public Library of Science |
record_format | MEDLINE/PubMed |
spelling | pubmed-34771542012-10-23 Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex Chadderdon, George L. Neymotin, Samuel A. Kerr, Cliff C. Lytton, William W. PLoS One Research Article Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (−1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior. Public Library of Science 2012-10-19 /pmc/articles/PMC3477154/ /pubmed/23094042 http://dx.doi.org/10.1371/journal.pone.0047251 Text en © 2012 Kerr et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited. |
spellingShingle | Research Article Chadderdon, George L. Neymotin, Samuel A. Kerr, Cliff C. Lytton, William W. Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex |
title | Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex |
title_full | Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex |
title_fullStr | Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex |
title_full_unstemmed | Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex |
title_short | Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex |
title_sort | reinforcement learning of targeted movement in a spiking neuronal model of motor cortex |
topic | Research Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477154/ https://www.ncbi.nlm.nih.gov/pubmed/23094042 http://dx.doi.org/10.1371/journal.pone.0047251 |
work_keys_str_mv | AT chadderdongeorgel reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex AT neymotinsamuela reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex AT kerrcliffc reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex AT lyttonwilliamw reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex |