Cargando…

Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL...

Descripción completa

Detalles Bibliográficos
Autores principales: Wärnberg, Emil, Kumar, Arvind
Formato: Online Artículo Texto
Lenguaje:English
Publicado: National Academy of Sciences 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10410740/
https://www.ncbi.nlm.nih.gov/pubmed/37527344
http://dx.doi.org/10.1073/pnas.2221994120
_version_ 1785086519066755072
author Wärnberg, Emil
Kumar, Arvind
author_facet Wärnberg, Emil
Kumar, Arvind
author_sort Wärnberg, Emil
collection PubMed
description It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, “three-factor” corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine.
format Online
Article
Text
id pubmed-10410740
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher National Academy of Sciences
record_format MEDLINE/PubMed
spelling pubmed-104107402023-08-10 Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia Wärnberg, Emil Kumar, Arvind Proc Natl Acad Sci U S A Biological Sciences It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, “three-factor” corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine. National Academy of Sciences 2023-08-01 2023-08-08 /pmc/articles/PMC10410740/ /pubmed/37527344 http://dx.doi.org/10.1073/pnas.2221994120 Text en Copyright © 2023 the Author(s). Published by PNAS. https://creativecommons.org/licenses/by/4.0/This open access article is distributed under Creative Commons Attribution License 4.0 (CC BY) (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Biological Sciences
Wärnberg, Emil
Kumar, Arvind
Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
title Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
title_full Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
title_fullStr Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
title_full_unstemmed Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
title_short Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
title_sort feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
topic Biological Sciences
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10410740/
https://www.ncbi.nlm.nih.gov/pubmed/37527344
http://dx.doi.org/10.1073/pnas.2221994120
work_keys_str_mv AT warnbergemil feasibilityofdopamineasavectorvaluedfeedbacksignalinthebasalganglia
AT kumararvind feasibilityofdopamineasavectorvaluedfeedbacksignalinthebasalganglia