Cargando…

Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them

When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variab...

Descripción completa

Detalles Bibliográficos
Autores principales: van Mastrigt, Nina M., van der Kooij, Katinka, Smeets, Jeroen B. J.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer Berlin Heidelberg 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8382626/
https://www.ncbi.nlm.nih.gov/pubmed/34341885
http://dx.doi.org/10.1007/s00422-021-00884-8
_version_ 1783741572641193984
author van Mastrigt, Nina M.
van der Kooij, Katinka
Smeets, Jeroen B. J.
author_facet van Mastrigt, Nina M.
van der Kooij, Katinka
Smeets, Jeroen B. J.
author_sort van Mastrigt, Nina M.
collection PubMed
description When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variability can be estimated from differences between subsequent movements. Can one estimate exploration reliably from such trial-to-trial changes when studying reward-based motor learning? To answer this question, we tried to reconstruct the exploration underlying learning as described by four existing reward-based motor learning models. We simulated learning for various learner and task characteristics. If we simply determined the additional change post-failure, estimates of exploration were sensitive to learner and task characteristics. We identified two pitfalls in quantifying exploration based on trial-to-trial changes. Firstly, performance-dependent feedback can cause correlated samples of motor noise and exploration on successful trials, which biases exploration estimates. Secondly, the trial relative to which trial-to-trial change is calculated may also contain exploration, which causes underestimation. As a solution, we developed the additional trial-to-trial change (ATTC) method. By moving the reference trial one trial back and subtracting trial-to-trial changes following specific sequences of trial outcomes, exploration can be estimated reliably for the three models that explore based on the outcome of only the previous trial. Since ATTC estimates are based on a selection of trial sequences, this method requires many trials. In conclusion, if exploration is a binary function of previous trial outcome, the ATTC method allows for a model-free quantification of exploration. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00422-021-00884-8.
format Online
Article
Text
id pubmed-8382626
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Springer Berlin Heidelberg
record_format MEDLINE/PubMed
spelling pubmed-83826262021-09-09 Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them van Mastrigt, Nina M. van der Kooij, Katinka Smeets, Jeroen B. J. Biol Cybern Original Article When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variability can be estimated from differences between subsequent movements. Can one estimate exploration reliably from such trial-to-trial changes when studying reward-based motor learning? To answer this question, we tried to reconstruct the exploration underlying learning as described by four existing reward-based motor learning models. We simulated learning for various learner and task characteristics. If we simply determined the additional change post-failure, estimates of exploration were sensitive to learner and task characteristics. We identified two pitfalls in quantifying exploration based on trial-to-trial changes. Firstly, performance-dependent feedback can cause correlated samples of motor noise and exploration on successful trials, which biases exploration estimates. Secondly, the trial relative to which trial-to-trial change is calculated may also contain exploration, which causes underestimation. As a solution, we developed the additional trial-to-trial change (ATTC) method. By moving the reference trial one trial back and subtracting trial-to-trial changes following specific sequences of trial outcomes, exploration can be estimated reliably for the three models that explore based on the outcome of only the previous trial. Since ATTC estimates are based on a selection of trial sequences, this method requires many trials. In conclusion, if exploration is a binary function of previous trial outcome, the ATTC method allows for a model-free quantification of exploration. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s00422-021-00884-8. Springer Berlin Heidelberg 2021-08-02 2021 /pmc/articles/PMC8382626/ /pubmed/34341885 http://dx.doi.org/10.1007/s00422-021-00884-8 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Original Article
van Mastrigt, Nina M.
van der Kooij, Katinka
Smeets, Jeroen B. J.
Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
title Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
title_full Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
title_fullStr Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
title_full_unstemmed Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
title_short Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
title_sort pitfalls in quantifying exploration in reward-based motor learning and how to avoid them
topic Original Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8382626/
https://www.ncbi.nlm.nih.gov/pubmed/34341885
http://dx.doi.org/10.1007/s00422-021-00884-8
work_keys_str_mv AT vanmastrigtninam pitfallsinquantifyingexplorationinrewardbasedmotorlearningandhowtoavoidthem
AT vanderkooijkatinka pitfallsinquantifyingexplorationinrewardbasedmotorlearningandhowtoavoidthem
AT smeetsjeroenbj pitfallsinquantifyingexplorationinrewardbasedmotorlearningandhowtoavoidthem