Cargando…

Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex

Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a...

Descripción completa

Detalles Bibliográficos
Autores principales: Chadderdon, George L., Neymotin, Samuel A., Kerr, Cliff C., Lytton, William W.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Public Library of Science 2012
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477154/
https://www.ncbi.nlm.nih.gov/pubmed/23094042
http://dx.doi.org/10.1371/journal.pone.0047251
_version_ 1782247205067816960
author Chadderdon, George L.
Neymotin, Samuel A.
Kerr, Cliff C.
Lytton, William W.
author_facet Chadderdon, George L.
Neymotin, Samuel A.
Kerr, Cliff C.
Lytton, William W.
author_sort Chadderdon, George L.
collection PubMed
description Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (−1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior.
format Online
Article
Text
id pubmed-3477154
institution National Center for Biotechnology Information
language English
publishDate 2012
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-34771542012-10-23 Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex Chadderdon, George L. Neymotin, Samuel A. Kerr, Cliff C. Lytton, William W. PLoS One Research Article Sensorimotor control has traditionally been considered from a control theory perspective, without relation to neurobiology. In contrast, here we utilized a spiking-neuron model of motor cortex and trained it to perform a simple movement task, which consisted of rotating a single-joint “forearm” to a target. Learning was based on a reinforcement mechanism analogous to that of the dopamine system. This provided a global reward or punishment signal in response to decreasing or increasing distance from hand to target, respectively. Output was partially driven by Poisson motor babbling, creating stochastic movements that could then be shaped by learning. The virtual forearm consisted of a single segment rotated around an elbow joint, controlled by flexor and extensor muscles. The model consisted of 144 excitatory and 64 inhibitory event-based neurons, each with AMPA, NMDA, and GABA synapses. Proprioceptive cell input to this model encoded the 2 muscle lengths. Plasticity was only enabled in feedforward connections between input and output excitatory units, using spike-timing-dependent eligibility traces for synaptic credit or blame assignment. Learning resulted from a global 3-valued signal: reward (+1), no learning (0), or punishment (−1), corresponding to phasic increases, lack of change, or phasic decreases of dopaminergic cell firing, respectively. Successful learning only occurred when both reward and punishment were enabled. In this case, 5 target angles were learned successfully within 180 s of simulation time, with a median error of 8 degrees. Motor babbling allowed exploratory learning, but decreased the stability of the learned behavior, since the hand continued moving after reaching the target. Our model demonstrated that a global reinforcement signal, coupled with eligibility traces for synaptic plasticity, can train a spiking sensorimotor network to perform goal-directed motor behavior. Public Library of Science 2012-10-19 /pmc/articles/PMC3477154/ /pubmed/23094042 http://dx.doi.org/10.1371/journal.pone.0047251 Text en © 2012 Kerr et al http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited.
spellingShingle Research Article
Chadderdon, George L.
Neymotin, Samuel A.
Kerr, Cliff C.
Lytton, William W.
Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
title Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
title_full Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
title_fullStr Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
title_full_unstemmed Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
title_short Reinforcement Learning of Targeted Movement in a Spiking Neuronal Model of Motor Cortex
title_sort reinforcement learning of targeted movement in a spiking neuronal model of motor cortex
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3477154/
https://www.ncbi.nlm.nih.gov/pubmed/23094042
http://dx.doi.org/10.1371/journal.pone.0047251
work_keys_str_mv AT chadderdongeorgel reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex
AT neymotinsamuela reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex
AT kerrcliffc reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex
AT lyttonwilliamw reinforcementlearningoftargetedmovementinaspikingneuronalmodelofmotorcortex