Cargando…

Retrospective model-based inference guides model-free credit assignment

An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an intera...

Descripción completa

Detalles Bibliográficos
Autores principales: Moran, Rani, Keramati, Mehdi, Dayan, Peter, Dolan, Raymond J.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Nature Publishing Group UK 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6375980/
https://www.ncbi.nlm.nih.gov/pubmed/30765718
http://dx.doi.org/10.1038/s41467-019-08662-8
_version_ 1783395466780606464
author Moran, Rani
Keramati, Mehdi
Dayan, Peter
Dolan, Raymond J.
author_facet Moran, Rani
Keramati, Mehdi
Dayan, Peter
Dolan, Raymond J.
author_sort Moran, Rani
collection PubMed
description An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants’ momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions.
format Online
Article
Text
id pubmed-6375980
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher Nature Publishing Group UK
record_format MEDLINE/PubMed
spelling pubmed-63759802019-02-19 Retrospective model-based inference guides model-free credit assignment Moran, Rani Keramati, Mehdi Dayan, Peter Dolan, Raymond J. Nat Commun Article An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants’ momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions. Nature Publishing Group UK 2019-02-14 /pmc/articles/PMC6375980/ /pubmed/30765718 http://dx.doi.org/10.1038/s41467-019-08662-8 Text en © The Author(s) 2019 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
spellingShingle Article
Moran, Rani
Keramati, Mehdi
Dayan, Peter
Dolan, Raymond J.
Retrospective model-based inference guides model-free credit assignment
title Retrospective model-based inference guides model-free credit assignment
title_full Retrospective model-based inference guides model-free credit assignment
title_fullStr Retrospective model-based inference guides model-free credit assignment
title_full_unstemmed Retrospective model-based inference guides model-free credit assignment
title_short Retrospective model-based inference guides model-free credit assignment
title_sort retrospective model-based inference guides model-free credit assignment
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6375980/
https://www.ncbi.nlm.nih.gov/pubmed/30765718
http://dx.doi.org/10.1038/s41467-019-08662-8
work_keys_str_mv AT moranrani retrospectivemodelbasedinferenceguidesmodelfreecreditassignment
AT keramatimehdi retrospectivemodelbasedinferenceguidesmodelfreecreditassignment
AT dayanpeter retrospectivemodelbasedinferenceguidesmodelfreecreditassignment
AT dolanraymondj retrospectivemodelbasedinferenceguidesmodelfreecreditassignment