Cargando…

You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning

Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study propose...

Descripción completa

Detalles Bibliográficos
Autores principales: Barros, Pablo, Bloem, Anne C., Hootsmans, Inge M., Opheij, Lena M., Toebosch, Romain H. A., Barakova, Emilia, Sciutti, Alessandra
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8323774/
https://www.ncbi.nlm.nih.gov/pubmed/34336935
http://dx.doi.org/10.3389/frobt.2021.669990
_version_ 1783731305773531136
author Barros, Pablo
Bloem, Anne C.
Hootsmans, Inge M.
Opheij, Lena M.
Toebosch, Romain H. A.
Barakova, Emilia
Sciutti, Alessandra
author_facet Barros, Pablo
Bloem, Anne C.
Hootsmans, Inge M.
Opheij, Lena M.
Toebosch, Romain H. A.
Barakova, Emilia
Sciutti, Alessandra
author_sort Barros, Pablo
collection PubMed
description Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef’s Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent’s actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef’s Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two “dummy” baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef’s Hat competitive game and the implementation of the Chef’s Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning.
format Online
Article
Text
id pubmed-8323774
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Frontiers Media S.A.
record_format MEDLINE/PubMed
spelling pubmed-83237742021-07-31 You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning Barros, Pablo Bloem, Anne C. Hootsmans, Inge M. Opheij, Lena M. Toebosch, Romain H. A. Barakova, Emilia Sciutti, Alessandra Front Robot AI Robotics and AI Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef’s Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent’s actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef’s Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two “dummy” baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef’s Hat competitive game and the implementation of the Chef’s Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning. Frontiers Media S.A. 2021-07-16 /pmc/articles/PMC8323774/ /pubmed/34336935 http://dx.doi.org/10.3389/frobt.2021.669990 Text en Copyright © 2021 Barros, Bloem, Hootsmans, Opheij, Toebosch, Barakova and Sciutti. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
spellingShingle Robotics and AI
Barros, Pablo
Bloem, Anne C.
Hootsmans, Inge M.
Opheij, Lena M.
Toebosch, Romain H. A.
Barakova, Emilia
Sciutti, Alessandra
You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
title You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
title_full You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
title_fullStr You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
title_full_unstemmed You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
title_short You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
title_sort you were always on my mind: introducing chef’s hat and copper for personalized reinforcement learning
topic Robotics and AI
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8323774/
https://www.ncbi.nlm.nih.gov/pubmed/34336935
http://dx.doi.org/10.3389/frobt.2021.669990
work_keys_str_mv AT barrospablo youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning
AT bloemannec youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning
AT hootsmansingem youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning
AT opheijlenam youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning
AT toeboschromainha youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning
AT barakovaemilia youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning
AT sciuttialessandra youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning