Cargando…
You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning
Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study propose...
Autores principales: | , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Frontiers Media S.A.
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8323774/ https://www.ncbi.nlm.nih.gov/pubmed/34336935 http://dx.doi.org/10.3389/frobt.2021.669990 |
_version_ | 1783731305773531136 |
---|---|
author | Barros, Pablo Bloem, Anne C. Hootsmans, Inge M. Opheij, Lena M. Toebosch, Romain H. A. Barakova, Emilia Sciutti, Alessandra |
author_facet | Barros, Pablo Bloem, Anne C. Hootsmans, Inge M. Opheij, Lena M. Toebosch, Romain H. A. Barakova, Emilia Sciutti, Alessandra |
author_sort | Barros, Pablo |
collection | PubMed |
description | Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef’s Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent’s actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef’s Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two “dummy” baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef’s Hat competitive game and the implementation of the Chef’s Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning. |
format | Online Article Text |
id | pubmed-8323774 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | Frontiers Media S.A. |
record_format | MEDLINE/PubMed |
spelling | pubmed-83237742021-07-31 You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning Barros, Pablo Bloem, Anne C. Hootsmans, Inge M. Opheij, Lena M. Toebosch, Romain H. A. Barakova, Emilia Sciutti, Alessandra Front Robot AI Robotics and AI Reinforcement learning simulation environments pose an important experimental test bed and facilitate data collection for developing AI-based robot applications. Most of them, however, focus on single-agent tasks, which limits their application to the development of social agents. This study proposes the Chef’s Hat simulation environment, which implements a multi-agent competitive card game that is a complete reproduction of the homonymous board game, designed to provoke competitive strategies in humans and emotional responses. The game was shown to be ideal for developing personalized reinforcement learning, in an online learning closed-loop scenario, as its state representation is extremely dynamic and directly related to each of the opponent’s actions. To adapt current reinforcement learning agents to this scenario, we also developed the COmPetitive Prioritized Experience Replay (COPPER) algorithm. With the help of COPPER and the Chef’s Hat simulation environment, we evaluated the following: (1) 12 experimental learning agents, trained via four different regimens (self-play, play against a naive baseline, PER, or COPPER) with three algorithms based on different state-of-the-art learning paradigms (PPO, DQN, and ACER), and two “dummy” baseline agents that take random actions, (2) the performance difference between COPPER and PER agents trained using the PPO algorithm and playing against different agents (PPO, DQN, and ACER) or all DQN agents, and (3) human performance when playing against two different collections of agents. Our experiments demonstrate that COPPER helps agents learn to adapt to different types of opponents, improving the performance when compared to off-line learning models. An additional contribution of the study is the formalization of the Chef’s Hat competitive game and the implementation of the Chef’s Hat Player Club, a collection of trained and assessed agents as an enabler for embedding human competitive strategies in social continual and competitive reinforcement learning. Frontiers Media S.A. 2021-07-16 /pmc/articles/PMC8323774/ /pubmed/34336935 http://dx.doi.org/10.3389/frobt.2021.669990 Text en Copyright © 2021 Barros, Bloem, Hootsmans, Opheij, Toebosch, Barakova and Sciutti. https://creativecommons.org/licenses/by/4.0/This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. |
spellingShingle | Robotics and AI Barros, Pablo Bloem, Anne C. Hootsmans, Inge M. Opheij, Lena M. Toebosch, Romain H. A. Barakova, Emilia Sciutti, Alessandra You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning |
title | You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning |
title_full | You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning |
title_fullStr | You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning |
title_full_unstemmed | You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning |
title_short | You Were Always on My Mind: Introducing Chef’s Hat and COPPER for Personalized Reinforcement Learning |
title_sort | you were always on my mind: introducing chef’s hat and copper for personalized reinforcement learning |
topic | Robotics and AI |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8323774/ https://www.ncbi.nlm.nih.gov/pubmed/34336935 http://dx.doi.org/10.3389/frobt.2021.669990 |
work_keys_str_mv | AT barrospablo youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning AT bloemannec youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning AT hootsmansingem youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning AT opheijlenam youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning AT toeboschromainha youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning AT barakovaemilia youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning AT sciuttialessandra youwerealwaysonmymindintroducingchefshatandcopperforpersonalizedreinforcementlearning |