Cargando…

Reinforcement Learning or Active Inference?

This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sam...

Descripción completa

Detalles Bibliográficos
Autores principales: Friston, Karl J., Daunizeau, Jean, Kiebel, Stefan J.
Formato: Texto
Lenguaje:English
Publicado: Public Library of Science 2009
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2713351/
https://www.ncbi.nlm.nih.gov/pubmed/19641614
http://dx.doi.org/10.1371/journal.pone.0006421
_version_ 1782169571940106240
author Friston, Karl J.
Daunizeau, Jean
Kiebel, Stefan J.
author_facet Friston, Karl J.
Daunizeau, Jean
Kiebel, Stefan J.
author_sort Friston, Karl J.
collection PubMed
description This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sampling of the environment to minimize their free-energy. Such agents learn causal structure in the environment and sample it in an adaptive and self-supervised fashion. This results in behavioural policies that reproduce those optimised by reinforcement learning and dynamic programming. Critically, we do not need to invoke the notion of reward, value or utility. We illustrate these points by solving a benchmark problem in dynamic programming; namely the mountain-car problem, using active perception or inference under the free-energy principle. The ensuing proof-of-concept may be important because the free-energy formulation furnishes a unified account of both action and perception and may speak to a reappraisal of the role of dopamine in the brain.
format Text
id pubmed-2713351
institution National Center for Biotechnology Information
language English
publishDate 2009
publisher Public Library of Science
record_format MEDLINE/PubMed
spelling pubmed-27133512009-07-28 Reinforcement Learning or Active Inference? Friston, Karl J. Daunizeau, Jean Kiebel, Stefan J. PLoS One Research Article This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sampling of the environment to minimize their free-energy. Such agents learn causal structure in the environment and sample it in an adaptive and self-supervised fashion. This results in behavioural policies that reproduce those optimised by reinforcement learning and dynamic programming. Critically, we do not need to invoke the notion of reward, value or utility. We illustrate these points by solving a benchmark problem in dynamic programming; namely the mountain-car problem, using active perception or inference under the free-energy principle. The ensuing proof-of-concept may be important because the free-energy formulation furnishes a unified account of both action and perception and may speak to a reappraisal of the role of dopamine in the brain. Public Library of Science 2009-07-29 /pmc/articles/PMC2713351/ /pubmed/19641614 http://dx.doi.org/10.1371/journal.pone.0006421 Text en Friston et al. http://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly credited.
spellingShingle Research Article
Friston, Karl J.
Daunizeau, Jean
Kiebel, Stefan J.
Reinforcement Learning or Active Inference?
title Reinforcement Learning or Active Inference?
title_full Reinforcement Learning or Active Inference?
title_fullStr Reinforcement Learning or Active Inference?
title_full_unstemmed Reinforcement Learning or Active Inference?
title_short Reinforcement Learning or Active Inference?
title_sort reinforcement learning or active inference?
topic Research Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2713351/
https://www.ncbi.nlm.nih.gov/pubmed/19641614
http://dx.doi.org/10.1371/journal.pone.0006421
work_keys_str_mv AT fristonkarlj reinforcementlearningoractiveinference
AT daunizeaujean reinforcementlearningoractiveinference
AT kiebelstefanj reinforcementlearningoractiveinference