Cargando…
Retrospective model-based inference guides model-free credit assignment
An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an intera...
Autores principales: | , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Nature Publishing Group UK
2019
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6375980/ https://www.ncbi.nlm.nih.gov/pubmed/30765718 http://dx.doi.org/10.1038/s41467-019-08662-8 |
_version_ | 1783395466780606464 |
---|---|
author | Moran, Rani Keramati, Mehdi Dayan, Peter Dolan, Raymond J. |
author_facet | Moran, Rani Keramati, Mehdi Dayan, Peter Dolan, Raymond J. |
author_sort | Moran, Rani |
collection | PubMed |
description | An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants’ momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions. |
format | Online Article Text |
id | pubmed-6375980 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2019 |
publisher | Nature Publishing Group UK |
record_format | MEDLINE/PubMed |
spelling | pubmed-63759802019-02-19 Retrospective model-based inference guides model-free credit assignment Moran, Rani Keramati, Mehdi Dayan, Peter Dolan, Raymond J. Nat Commun Article An extensive reinforcement learning literature shows that organisms assign credit efficiently, even under conditions of state uncertainty. However, little is known about credit-assignment when state uncertainty is subsequently resolved. Here, we address this problem within the framework of an interaction between model-free (MF) and model-based (MB) control systems. We present and support experimentally a theory of MB retrospective-inference. Within this framework, a MB system resolves uncertainty that prevailed when actions were taken thus guiding an MF credit-assignment. Using a task in which there was initial uncertainty about the lotteries that were chosen, we found that when participants’ momentary uncertainty about which lottery had generated an outcome was resolved by provision of subsequent information, participants preferentially assigned credit within a MF system to the lottery they retrospectively inferred was responsible for this outcome. These findings extend our knowledge about the range of MB functions and the scope of system interactions. Nature Publishing Group UK 2019-02-14 /pmc/articles/PMC6375980/ /pubmed/30765718 http://dx.doi.org/10.1038/s41467-019-08662-8 Text en © The Author(s) 2019 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. |
spellingShingle | Article Moran, Rani Keramati, Mehdi Dayan, Peter Dolan, Raymond J. Retrospective model-based inference guides model-free credit assignment |
title | Retrospective model-based inference guides model-free credit assignment |
title_full | Retrospective model-based inference guides model-free credit assignment |
title_fullStr | Retrospective model-based inference guides model-free credit assignment |
title_full_unstemmed | Retrospective model-based inference guides model-free credit assignment |
title_short | Retrospective model-based inference guides model-free credit assignment |
title_sort | retrospective model-based inference guides model-free credit assignment |
topic | Article |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6375980/ https://www.ncbi.nlm.nih.gov/pubmed/30765718 http://dx.doi.org/10.1038/s41467-019-08662-8 |
work_keys_str_mv | AT moranrani retrospectivemodelbasedinferenceguidesmodelfreecreditassignment AT keramatimehdi retrospectivemodelbasedinferenceguidesmodelfreecreditassignment AT dayanpeter retrospectivemodelbasedinferenceguidesmodelfreecreditassignment AT dolanraymondj retrospectivemodelbasedinferenceguidesmodelfreecreditassignment |