Cargando…

Offline replay supports planning in human reinforcement learning

Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether ‘offline’ integration during replay...

Descripción completa

Detalles Bibliográficos
Autores principales: Momennejad, Ida, Otto, A Ross, Daw, Nathaniel D, Norman, Kenneth A
Formato: Online Artículo Texto
Lenguaje:English
Publicado: eLife Sciences Publications, Ltd 2018
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6303108/
https://www.ncbi.nlm.nih.gov/pubmed/30547886
http://dx.doi.org/10.7554/eLife.32548
_version_ 1783382119532199936
author Momennejad, Ida
Otto, A Ross
Daw, Nathaniel D
Norman, Kenneth A
author_facet Momennejad, Ida
Otto, A Ross
Daw, Nathaniel D
Norman, Kenneth A
author_sort Momennejad, Ida
collection PubMed
description Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether ‘offline’ integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.
format Online
Article
Text
id pubmed-6303108
institution National Center for Biotechnology Information
language English
publishDate 2018
publisher eLife Sciences Publications, Ltd
record_format MEDLINE/PubMed
spelling pubmed-63031082019-01-04 Offline replay supports planning in human reinforcement learning Momennejad, Ida Otto, A Ross Daw, Nathaniel D Norman, Kenneth A eLife Neuroscience Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether ‘offline’ integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty. eLife Sciences Publications, Ltd 2018-12-14 /pmc/articles/PMC6303108/ /pubmed/30547886 http://dx.doi.org/10.7554/eLife.32548 Text en © 2018, Momennejad et al http://creativecommons.org/licenses/by/4.0/ http://creativecommons.org/licenses/by/4.0/This article is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/) , which permits unrestricted use and redistribution provided that the original author and source are credited.
spellingShingle Neuroscience
Momennejad, Ida
Otto, A Ross
Daw, Nathaniel D
Norman, Kenneth A
Offline replay supports planning in human reinforcement learning
title Offline replay supports planning in human reinforcement learning
title_full Offline replay supports planning in human reinforcement learning
title_fullStr Offline replay supports planning in human reinforcement learning
title_full_unstemmed Offline replay supports planning in human reinforcement learning
title_short Offline replay supports planning in human reinforcement learning
title_sort offline replay supports planning in human reinforcement learning
topic Neuroscience
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6303108/
https://www.ncbi.nlm.nih.gov/pubmed/30547886
http://dx.doi.org/10.7554/eLife.32548
work_keys_str_mv AT momennejadida offlinereplaysupportsplanninginhumanreinforcementlearning
AT ottoaross offlinereplaysupportsplanninginhumanreinforcementlearning
AT dawnathanield offlinereplaysupportsplanninginhumanreinforcementlearning
AT normankennetha offlinereplaysupportsplanninginhumanreinforcementlearning