Cargando…
Credit assignment to state-independent task representations and its relationship with model-based decision making
Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challe...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
National Academy of Sciences
2019
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6689934/ https://www.ncbi.nlm.nih.gov/pubmed/31320592 http://dx.doi.org/10.1073/pnas.1821647116 |
_version_ | 1783443117649690624 |
---|---|
author | Shahar, Nitzan Moran, Rani Hauser, Tobias U. Kievit, Rogier A. McNamee, Daniel Moutoussis, Michael Dolan, Raymond J. |
author_facet | Shahar, Nitzan Moran, Rani Hauser, Tobias U. Kievit, Rogier A. McNamee, Daniel Moutoussis, Michael Dolan, Raymond J. |
author_sort | Shahar, Nitzan |
collection | PubMed |
description | Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment. |
format | Online Article Text |
id | pubmed-6689934 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2019 |
publisher | National Academy of Sciences |
record_format | MEDLINE/PubMed |
spelling | pubmed-66899342019-08-14 Credit assignment to state-independent task representations and its relationship with model-based decision making Shahar, Nitzan Moran, Rani Hauser, Tobias U. Kievit, Rogier A. McNamee, Daniel Moutoussis, Michael Dolan, Raymond J. Proc Natl Acad Sci U S A Social Sciences Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment. National Academy of Sciences 2019-08-06 2019-07-18 /pmc/articles/PMC6689934/ /pubmed/31320592 http://dx.doi.org/10.1073/pnas.1821647116 Text en Copyright © 2019 the Author(s). Published by PNAS. http://creativecommons.org/licenses/by/4.0/ https://creativecommons.org/licenses/by/4.0/This open access article is distributed under Creative Commons Attribution License 4.0 (CC BY) (http://creativecommons.org/licenses/by/4.0/) . |
spellingShingle | Social Sciences Shahar, Nitzan Moran, Rani Hauser, Tobias U. Kievit, Rogier A. McNamee, Daniel Moutoussis, Michael Dolan, Raymond J. Credit assignment to state-independent task representations and its relationship with model-based decision making |
title | Credit assignment to state-independent task representations and its relationship with model-based decision making |
title_full | Credit assignment to state-independent task representations and its relationship with model-based decision making |
title_fullStr | Credit assignment to state-independent task representations and its relationship with model-based decision making |
title_full_unstemmed | Credit assignment to state-independent task representations and its relationship with model-based decision making |
title_short | Credit assignment to state-independent task representations and its relationship with model-based decision making |
title_sort | credit assignment to state-independent task representations and its relationship with model-based decision making |
topic | Social Sciences |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6689934/ https://www.ncbi.nlm.nih.gov/pubmed/31320592 http://dx.doi.org/10.1073/pnas.1821647116 |
work_keys_str_mv | AT shaharnitzan creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT moranrani creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT hausertobiasu creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT kievitrogiera creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT mcnameedaniel creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT moutoussismichael creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking AT dolanraymondj creditassignmenttostateindependenttaskrepresentationsanditsrelationshipwithmodelbaseddecisionmaking |