Cargando…

Using cognitive psychology to understand GPT-3

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavio...

Descripción completa

Detalles Bibliográficos
Autores principales: Binz, Marcel, Schulz, Eric
Formato: Online Artículo Texto
Lenguaje:English
Publicado: National Academy of Sciences 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9963545/
https://www.ncbi.nlm.nih.gov/pubmed/36730192
http://dx.doi.org/10.1073/pnas.2218523120
_version_ 1784896278151299072
author Binz, Marcel
Schulz, Eric
author_facet Binz, Marcel
Schulz, Eric
author_sort Binz, Marcel
collection PubMed
description We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.
format Online
Article
Text
id pubmed-9963545
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher National Academy of Sciences
record_format MEDLINE/PubMed
spelling pubmed-99635452023-08-02 Using cognitive psychology to understand GPT-3 Binz, Marcel Schulz, Eric Proc Natl Acad Sci U S A Social Sciences We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents. National Academy of Sciences 2023-02-02 2023-02-07 /pmc/articles/PMC9963545/ /pubmed/36730192 http://dx.doi.org/10.1073/pnas.2218523120 Text en Copyright © 2023 the Author(s). Published by PNAS. https://creativecommons.org/licenses/by-nc-nd/4.0/This article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND) (https://creativecommons.org/licenses/by-nc-nd/4.0/) .
spellingShingle Social Sciences
Binz, Marcel
Schulz, Eric
Using cognitive psychology to understand GPT-3
title Using cognitive psychology to understand GPT-3
title_full Using cognitive psychology to understand GPT-3
title_fullStr Using cognitive psychology to understand GPT-3
title_full_unstemmed Using cognitive psychology to understand GPT-3
title_short Using cognitive psychology to understand GPT-3
title_sort using cognitive psychology to understand gpt-3
topic Social Sciences
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9963545/
https://www.ncbi.nlm.nih.gov/pubmed/36730192
http://dx.doi.org/10.1073/pnas.2218523120
work_keys_str_mv AT binzmarcel usingcognitivepsychologytounderstandgpt3
AT schulzeric usingcognitivepsychologytounderstandgpt3