Cargando…

Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under...

Descripción completa

Detalles Bibliográficos
Autores principales: Trella, Anna L., Zhang, Kelly W., Nahum-Shani, Inbal, Shetty, Vivek, Doshi-Velez, Finale, Murphy, Susan A.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9881427/
https://www.ncbi.nlm.nih.gov/pubmed/36713810
http://dx.doi.org/10.3390/a15080255
_version_ 1784879105182793728
author Trella, Anna L.
Zhang, Kelly W.
Nahum-Shani, Inbal
Shetty, Vivek
Doshi-Velez, Finale
Murphy, Susan A.
author_facet Trella, Anna L.
Zhang, Kelly W.
Nahum-Shani, Inbal
Shetty, Vivek
Doshi-Velez, Finale
Murphy, Susan A.
author_sort Trella, Anna L.
collection PubMed
description Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users’ tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.
format Online
Article
Text
id pubmed-9881427
institution National Center for Biotechnology Information
language English
publishDate 2022
record_format MEDLINE/PubMed
spelling pubmed-98814272023-01-27 Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines Trella, Anna L. Zhang, Kelly W. Nahum-Shani, Inbal Shetty, Vivek Doshi-Velez, Finale Murphy, Susan A. Algorithms Article Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (predictability, computability, stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning to the design of RL algorithms for the digital interventions setting. Furthermore, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We show how we used the PCS framework to design an RL algorithm for Oralytics, a mobile health study aiming to improve users’ tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022. 2022-08 2022-07-22 /pmc/articles/PMC9881427/ /pubmed/36713810 http://dx.doi.org/10.3390/a15080255 Text en https://creativecommons.org/licenses/by/4.0/This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
spellingShingle Article
Trella, Anna L.
Zhang, Kelly W.
Nahum-Shani, Inbal
Shetty, Vivek
Doshi-Velez, Finale
Murphy, Susan A.
Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
title Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
title_full Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
title_fullStr Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
title_full_unstemmed Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
title_short Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-Implementation Guidelines
title_sort designing reinforcement learning algorithms for digital interventions: pre-implementation guidelines
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9881427/
https://www.ncbi.nlm.nih.gov/pubmed/36713810
http://dx.doi.org/10.3390/a15080255
work_keys_str_mv AT trellaannal designingreinforcementlearningalgorithmsfordigitalinterventionspreimplementationguidelines
AT zhangkellyw designingreinforcementlearningalgorithmsfordigitalinterventionspreimplementationguidelines
AT nahumshaniinbal designingreinforcementlearningalgorithmsfordigitalinterventionspreimplementationguidelines
AT shettyvivek designingreinforcementlearningalgorithmsfordigitalinterventionspreimplementationguidelines
AT doshivelezfinale designingreinforcementlearningalgorithmsfordigitalinterventionspreimplementationguidelines
AT murphysusana designingreinforcementlearningalgorithmsfordigitalinterventionspreimplementationguidelines