Cargando…

Policy Adjustment in a Dynamic Economic Game

Making sequential decisions to harvest rewards is a notoriously difficult problem. One difficulty is that the real world is not stationary and the reward expected from a contemplated action may depend in complex ways on the history of an animal's choices. Previous functional neuroimaging work c...

Descripción completa

Detalles Bibliográficos
Autores principales: Li, Jian, McClure, Samuel M., King-Casas, Brooks, Read Montague, P.
Formato: Texto
Lenguaje:English
Publicado: Public Library of Science 2006
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1762366/
https://www.ncbi.nlm.nih.gov/pubmed/17183636
http://dx.doi.org/10.1371/journal.pone.0000103
_version_ 1782131552081149952
author Li, Jian
McClure, Samuel M.
King-Casas, Brooks
Read Montague, P.
author_facet Li, Jian
McClure, Samuel M.
King-Casas, Brooks
Read Montague, P.
author_sort Li, Jian
collection PubMed
description Making sequential decisions to harvest rewards is a notoriously difficult problem. One difficulty is that the real world is not stationary and the reward expected from a contemplated action may depend in complex ways on the history of an animal's choices. Previous functional neuroimaging work combined with principled models has detected brain responses that correlate with computations thought to guide simple learning and action choice. Those works generally employed instrumental conditioning tasks with fixed action-reward contingencies. For real-world learning problems, the history of reward-harvesting choices can change the likelihood of rewards collected by the same choices in the near-term future. We used functional MRI to probe brain and behavioral responses in a continuous decision-making task where reward contingency is a function of both a subject's immediate choice and his choice history. In these more complex tasks, we demonstrated that a simple actor-critic model can account for both the subjects' behavioral and brain responses, and identified a reward prediction error signal in ventral striatal structures active during these non-stationary decision tasks. However, a sudden introduction of new reward structures engages more complex control circuitry in the prefrontal cortex (inferior frontal gyrus and anterior insula) and is not captured by a simple actor-critic model. Taken together, these results extend our knowledge of reward-learning signals into more complex, history-dependent choice tasks. They also highlight the important interplay between striatum and prefrontal cortex as decision-makers respond to the strategic demands imposed by non-stationary reward environments more reminiscent of real-world tasks.
format Text
id pubmed-1762366
institution National Center for Biotechnology Information
language English
publishDate 2006
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-17623662007-03-06 Policy Adjustment in a Dynamic Economic Game Li, Jian McClure, Samuel M. King-Casas, Brooks Read Montague, P. PLoS One Research Article Making sequential decisions to harvest rewards is a notoriously difficult problem. One difficulty is that the real world is not stationary and the reward expected from a contemplated action may depend in complex ways on the history of an animal's choices. Previous functional neuroimaging work combined with principled models has detected brain responses that correlate with computations thought to guide simple learning and action choice. Those works generally employed instrumental conditioning tasks with fixed action-reward contingencies. For real-world learning problems, the history of reward-harvesting choices can change the likelihood of rewards collected by the same choices in the near-term future. We used functional MRI to probe brain and behavioral responses in a continuous decision-making task where reward contingency is a function of both a subject's immediate choice and his choice history. In these more complex tasks, we demonstrated that a simple actor-critic model can account for both the subjects' behavioral and brain responses, and identified a reward prediction error signal in ventral striatal structures active during these non-stationary decision tasks. However, a sudden introduction of new reward structures engages more complex control circuitry in the prefrontal cortex (inferior frontal gyrus and anterior insula) and is not captured by a simple actor-critic model. Taken together, these results extend our knowledge of reward-learning signals into more complex, history-dependent choice tasks. They also highlight the important interplay between striatum and prefrontal cortex as decision-makers respond to the strategic demands imposed by non-stationary reward environments more reminiscent of real-world tasks. Public Library of Science 2006-12-20 /pmc/articles/PMC1762366/ /pubmed/17183636 http://dx.doi.org/10.1371/journal.pone.0000103 Text en Li et al. http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited.
spellingShingle Research Article
Li, Jian
McClure, Samuel M.
King-Casas, Brooks
Read Montague, P.
Policy Adjustment in a Dynamic Economic Game
title Policy Adjustment in a Dynamic Economic Game
title_full Policy Adjustment in a Dynamic Economic Game
title_fullStr Policy Adjustment in a Dynamic Economic Game
title_full_unstemmed Policy Adjustment in a Dynamic Economic Game
title_short Policy Adjustment in a Dynamic Economic Game
title_sort policy adjustment in a dynamic economic game
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1762366/
https://www.ncbi.nlm.nih.gov/pubmed/17183636
http://dx.doi.org/10.1371/journal.pone.0000103
work_keys_str_mv AT lijian policyadjustmentinadynamiceconomicgame
AT mccluresamuelm policyadjustmentinadynamiceconomicgame
AT kingcasasbrooks policyadjustmentinadynamiceconomicgame
AT readmontaguep policyadjustmentinadynamiceconomicgame