Cargando…
A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care
BACKGROUND: Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation moni...
Autores principales: | , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
SAGE Publications
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10375039/ https://www.ncbi.nlm.nih.gov/pubmed/37790171 http://dx.doi.org/10.1177/26334895231187906 |
_version_ | 1785078932108738560 |
---|---|
author | Berkel, Cady Knox, Dillon C. Flemotomos, Nikolaos Martinez, Victor R. Atkins, David C. Narayanan, Shrikanth S. Rodriguez, Lizeth Alonso Gallo, Carlos G. Smith, Justin D. |
author_facet | Berkel, Cady Knox, Dillon C. Flemotomos, Nikolaos Martinez, Victor R. Atkins, David C. Narayanan, Shrikanth S. Rodriguez, Lizeth Alonso Gallo, Carlos G. Smith, Justin D. |
author_sort | Berkel, Cady |
collection | PubMed |
description | BACKGROUND: Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner. METHODS: In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes. RESULTS: Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83–1.02 to 0.62–0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81–27.3 to 0.62–19.50, resulting in an approximate average improvement of 18%. CONCLUSIONS: These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented. TRIAL REGISTRATION: NCT03013309 ClinicalTrials.gov. |
format | Online Article Text |
id | pubmed-10375039 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | SAGE Publications |
record_format | MEDLINE/PubMed |
spelling | pubmed-103750392023-10-03 A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care Berkel, Cady Knox, Dillon C. Flemotomos, Nikolaos Martinez, Victor R. Atkins, David C. Narayanan, Shrikanth S. Rodriguez, Lizeth Alonso Gallo, Carlos G. Smith, Justin D. Implement Res Pract Original Empirical Research BACKGROUND: Evidence-based parenting programs effectively prevent the onset and escalation of child and adolescent behavioral health problems. When programs have been taken to scale, declines in the quality of implementation diminish intervention effects. Gold-standard methods of implementation monitoring are cost-prohibitive and impractical in resource-scarce delivery systems. Technological developments using computational linguistics and machine learning offer an opportunity to assess fidelity in a low burden, timely, and comprehensive manner. METHODS: In this study, we test two natural language processing (NLP) methods [i.e., Term Frequency-Inverse Document Frequency (TF-IDF) and Bidirectional Encoder Representations from Transformers (BERT)] to assess the delivery of the Family Check-Up 4 Health (FCU4Health) program in a type 2 hybrid effectiveness-implementation trial conducted in primary care settings that serve primarily Latino families. We trained and evaluated models using 116 English and 81 Spanish-language transcripts from the 113 families who initiated FCU4Health services. We evaluated the concurrent validity of the TF-IDF and BERT models using observer ratings of program sessions using the COACH measure of competent adherence. Following the Implementation Cascade model, we assessed predictive validity using multiple indicators of parent engagement, which have been demonstrated to predict improvements in parenting and child outcomes. RESULTS: Both TF-IDF and BERT ratings were significantly associated with observer ratings and engagement outcomes. Using mean squared error, results demonstrated improvement over baseline for observer ratings from a range of 0.83–1.02 to 0.62–0.76, resulting in an average improvement of 24%. Similarly, results demonstrated improvement over baseline for parent engagement indicators from a range of 0.81–27.3 to 0.62–19.50, resulting in an approximate average improvement of 18%. CONCLUSIONS: These results demonstrate the potential for NLP methods to assess implementation in evidence-based parenting programs delivered at scale. Future directions are presented. TRIAL REGISTRATION: NCT03013309 ClinicalTrials.gov. SAGE Publications 2023-07-25 /pmc/articles/PMC10375039/ /pubmed/37790171 http://dx.doi.org/10.1177/26334895231187906 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by-nc/4.0/This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage). |
spellingShingle | Original Empirical Research Berkel, Cady Knox, Dillon C. Flemotomos, Nikolaos Martinez, Victor R. Atkins, David C. Narayanan, Shrikanth S. Rodriguez, Lizeth Alonso Gallo, Carlos G. Smith, Justin D. A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
title | A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
title_full | A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
title_fullStr | A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
title_full_unstemmed | A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
title_short | A machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
title_sort | machine learning approach to improve implementation monitoring of family-based preventive interventions in primary care |
topic | Original Empirical Research |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10375039/ https://www.ncbi.nlm.nih.gov/pubmed/37790171 http://dx.doi.org/10.1177/26334895231187906 |
work_keys_str_mv | AT berkelcady amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT knoxdillonc amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT flemotomosnikolaos amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT martinezvictorr amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT atkinsdavidc amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT narayananshrikanths amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT rodriguezlizethalonso amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT gallocarlosg amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT smithjustind amachinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT berkelcady machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT knoxdillonc machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT flemotomosnikolaos machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT martinezvictorr machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT atkinsdavidc machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT narayananshrikanths machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT rodriguezlizethalonso machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT gallocarlosg machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare AT smithjustind machinelearningapproachtoimproveimplementationmonitoringoffamilybasedpreventiveinterventionsinprimarycare |