Cargando…
Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia
It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
National Academy of Sciences
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10410740/ https://www.ncbi.nlm.nih.gov/pubmed/37527344 http://dx.doi.org/10.1073/pnas.2221994120 |
_version_ | 1785086519066755072 |
---|---|
author | Wärnberg, Emil Kumar, Arvind |
author_facet | Wärnberg, Emil Kumar, Arvind |
author_sort | Wärnberg, Emil |
collection | PubMed |
description | It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, “three-factor” corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine. |
format | Online Article Text |
id | pubmed-10410740 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | National Academy of Sciences |
record_format | MEDLINE/PubMed |
spelling | pubmed-104107402023-08-10 Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia Wärnberg, Emil Kumar, Arvind Proc Natl Acad Sci U S A Biological Sciences It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, “three-factor” corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine. National Academy of Sciences 2023-08-01 2023-08-08 /pmc/articles/PMC10410740/ /pubmed/37527344 http://dx.doi.org/10.1073/pnas.2221994120 Text en Copyright © 2023 the Author(s). Published by PNAS. https://creativecommons.org/licenses/by/4.0/This open access article is distributed under Creative Commons Attribution License 4.0 (CC BY) (https://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Biological Sciences Wärnberg, Emil Kumar, Arvind Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
title | Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
title_full | Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
title_fullStr | Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
title_full_unstemmed | Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
title_short | Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
title_sort | feasibility of dopamine as a vector-valued feedback signal in the basal ganglia |
topic | Biological Sciences |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10410740/ https://www.ncbi.nlm.nih.gov/pubmed/37527344 http://dx.doi.org/10.1073/pnas.2221994120 |
work_keys_str_mv | AT warnbergemil feasibilityofdopamineasavectorvaluedfeedbacksignalinthebasalganglia AT kumararvind feasibilityofdopamineasavectorvaluedfeedbacksignalinthebasalganglia |