Cargando…

A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks

Early detection of risk of failure on interactive tasks comes with great potential for better understanding how examinees differ in their initial behavior as well as for adaptively tailoring interactive tasks to examinees’ competence levels. Drawing on procedures originating in shopper intent predic...

Descripción completa

Detalles Bibliográficos
Autores principales: Ulitzsch, Esther, Ulitzsch, Vincent, He, Qiwei, Lüdtke, Oliver
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer US 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10125949/
https://www.ncbi.nlm.nih.gov/pubmed/35650385
http://dx.doi.org/10.3758/s13428-022-01844-1
_version_ 1785030131483410432
author Ulitzsch, Esther
Ulitzsch, Vincent
He, Qiwei
Lüdtke, Oliver
author_facet Ulitzsch, Esther
Ulitzsch, Vincent
He, Qiwei
Lüdtke, Oliver
author_sort Ulitzsch, Esther
collection PubMed
description Early detection of risk of failure on interactive tasks comes with great potential for better understanding how examinees differ in their initial behavior as well as for adaptively tailoring interactive tasks to examinees’ competence levels. Drawing on procedures originating in shopper intent prediction on e-commerce platforms, we introduce and showcase a machine learning-based procedure that leverages early-window clickstream data for systematically investigating early predictability of behavioral outcomes on interactive tasks. We derive features related to the occurrence, frequency, sequentiality, and timing of performed actions from early-window clickstreams and use extreme gradient boosting for classification. Multiple measures are suggested to evaluate the quality and utility of early predictions. The procedure is outlined by investigating early predictability of failure on two PIAAC 2012 Problem Solving in Technology Rich Environments (PSTRE) tasks. We investigated early windows of varying size in terms of time and in terms of actions. We achieved good prediction performance at stages where examinees had, on average, at least two thirds of their solution process ahead of them, and the vast majority of examinees who failed could potentially be detected to be at risk before completing the task. In-depth analyses revealed different features to be indicative of success and failure at different stages of the solution process, thereby highlighting the potential of the applied procedure for gaining a finer-grained understanding of the trajectories of behavioral patterns on interactive tasks.
format Online
Article
Text
id pubmed-10125949
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher Springer US
record_format MEDLINE/PubMed
spelling pubmed-101259492023-04-26 A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks Ulitzsch, Esther Ulitzsch, Vincent He, Qiwei Lüdtke, Oliver Behav Res Methods Article Early detection of risk of failure on interactive tasks comes with great potential for better understanding how examinees differ in their initial behavior as well as for adaptively tailoring interactive tasks to examinees’ competence levels. Drawing on procedures originating in shopper intent prediction on e-commerce platforms, we introduce and showcase a machine learning-based procedure that leverages early-window clickstream data for systematically investigating early predictability of behavioral outcomes on interactive tasks. We derive features related to the occurrence, frequency, sequentiality, and timing of performed actions from early-window clickstreams and use extreme gradient boosting for classification. Multiple measures are suggested to evaluate the quality and utility of early predictions. The procedure is outlined by investigating early predictability of failure on two PIAAC 2012 Problem Solving in Technology Rich Environments (PSTRE) tasks. We investigated early windows of varying size in terms of time and in terms of actions. We achieved good prediction performance at stages where examinees had, on average, at least two thirds of their solution process ahead of them, and the vast majority of examinees who failed could potentially be detected to be at risk before completing the task. In-depth analyses revealed different features to be indicative of success and failure at different stages of the solution process, thereby highlighting the potential of the applied procedure for gaining a finer-grained understanding of the trajectories of behavioral patterns on interactive tasks. Springer US 2022-06-01 2023 /pmc/articles/PMC10125949/ /pubmed/35650385 http://dx.doi.org/10.3758/s13428-022-01844-1 Text en © The Author(s) 2022 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Article
Ulitzsch, Esther
Ulitzsch, Vincent
He, Qiwei
Lüdtke, Oliver
A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
title A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
title_full A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
title_fullStr A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
title_full_unstemmed A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
title_short A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
title_sort machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10125949/
https://www.ncbi.nlm.nih.gov/pubmed/35650385
http://dx.doi.org/10.3758/s13428-022-01844-1
work_keys_str_mv AT ulitzschesther amachinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT ulitzschvincent amachinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT heqiwei amachinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT ludtkeoliver amachinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT ulitzschesther machinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT ulitzschvincent machinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT heqiwei machinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks
AT ludtkeoliver machinelearningbasedprocedureforleveragingclickstreamdatatoinvestigateearlypredictabilityoffailureoninteractivetasks