Cargando…

A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning

Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider ho...

Descripción completa

Detalles Bibliográficos
Autores principales: Franklin, Nicholas T, Frank, Michael J
Formato: Online Artículo Texto
Lenguaje:English
Publicado: eLife Sciences Publications, Ltd 2015
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4764588/
https://www.ncbi.nlm.nih.gov/pubmed/26705698
http://dx.doi.org/10.7554/eLife.12029
_version_ 1782417397988196352
author Franklin, Nicholas T
Frank, Michael J
author_facet Franklin, Nicholas T
Frank, Michael J
author_sort Franklin, Nicholas T
collection PubMed
description Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments. DOI: http://dx.doi.org/10.7554/eLife.12029.001
format Online
Article
Text
id pubmed-4764588
institution National Center for Biotechnology Information
language English
publishDate 2015
publisher eLife Sciences Publications, Ltd
record_format MEDLINE/PubMed
spelling pubmed-47645882016-02-25 A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning Franklin, Nicholas T Frank, Michael J eLife Computational and Systems Biology Convergent evidence suggests that the basal ganglia support reinforcement learning by adjusting action values according to reward prediction errors. However, adaptive behavior in stochastic environments requires the consideration of uncertainty to dynamically adjust the learning rate. We consider how cholinergic tonically active interneurons (TANs) may endow the striatum with such a mechanism in computational models spanning three Marr's levels of analysis. In the neural model, TANs modulate the excitability of spiny neurons, their population response to reinforcement, and hence the effective learning rate. Long TAN pauses facilitated robustness to spurious outcomes by increasing divergence in synaptic weights between neurons coding for alternative action values, whereas short TAN pauses facilitated stochastic behavior but increased responsiveness to change-points in outcome contingencies. A feedback control system allowed TAN pauses to be dynamically modulated by uncertainty across the spiny neuron population, allowing the system to self-tune and optimize performance across stochastic environments. DOI: http://dx.doi.org/10.7554/eLife.12029.001 eLife Sciences Publications, Ltd 2015-12-25 /pmc/articles/PMC4764588/ /pubmed/26705698 http://dx.doi.org/10.7554/eLife.12029 Text en © 2015, Franklin et al http://creativecommons.org/licenses/by/4.0/ This article is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use and redistribution provided that the original author and source are credited.
spellingShingle Computational and Systems Biology
Franklin, Nicholas T
Frank, Michael J
A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
title A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
title_full A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
title_fullStr A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
title_full_unstemmed A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
title_short A cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
title_sort cholinergic feedback circuit to regulate striatal population uncertainty and optimize reinforcement learning
topic Computational and Systems Biology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4764588/
https://www.ncbi.nlm.nih.gov/pubmed/26705698
http://dx.doi.org/10.7554/eLife.12029
work_keys_str_mv AT franklinnicholast acholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning
AT frankmichaelj acholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning
AT franklinnicholast cholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning
AT frankmichaelj cholinergicfeedbackcircuittoregulatestriatalpopulationuncertaintyandoptimizereinforcementlearning