Cargando…

Automatic rating of therapist facilitative interpersonal skills in text: A natural language processing application

BACKGROUND: While message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality–The Facilitative Interpersonal...

Descripción completa

Detalles Bibliográficos
Autores principales: Zech, James M., Steele, Robert, Foley, Victoria K., Hull, Thomas D.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Frontiers Media S.A. 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9425293/
https://www.ncbi.nlm.nih.gov/pubmed/36052318
http://dx.doi.org/10.3389/fdgth.2022.917918
Descripción
Sumario:BACKGROUND: While message-based therapy has been shown to be effective in treating a range of mood disorders, it is critical to ensure that providers are meeting a consistently high standard of care over this medium. One recently developed measure of messaging quality–The Facilitative Interpersonal Skills Task for Text (FIS-T)–provides estimates of therapists’ demonstrated ability to convey psychotherapy's common factors (e.g., hopefulness, warmth, persuasiveness) over text. However, the FIS-T's scoring procedure relies on trained human coders to manually code responses, thereby rendering the FIS-T an unscalable quality control tool for large messaging therapy platforms. OBJECTIVE: In the present study, researchers developed two algorithms to automatically score therapist performance on the FIS-T task. METHODS: The FIS-T was administered to 978 messaging therapists, whose responses were then manually scored by a trained team of raters. Two machine learning algorithms were then trained on task-taker messages and coder scores: a support vector regressor (SVR) and a transformer-based neural network (DistilBERT). RESULTS: The DistilBERT model had superior performance on the prediction task while providing a distribution of ratings that was more closely aligned with those of human raters, versus SVR. Specifically, the DistilBERT model was able to explain 58.8% of the variance (R(2 )= 0.588) in human-derived ratings and realized a prediction mean absolute error of 0.134 on a 1–5 scale. CONCLUSIONS: Algorithms can be effectively used to ensure that digital providers meet a consistently high standard of interactions in the course of messaging therapy. Natural language processing can be applied to develop new quality assurance systems in message-based digital psychotherapy.