Cargando…

The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability

BACKGROUND: Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field l...

Descripción completa

Detalles Bibliográficos
Autores principales: Lyon, Aaron R., Coifman, Jessica, Cook, Heather, McRee, Erin, Liu, Freda F., Ludwig, Kristy, Dorsey, Shannon, Koerner, Kelly, Munson, Sean A., McCauley, Elizabeth
Formato: Online Artículo Texto
Lenguaje:English
Publicado: BioMed Central 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8285864/
https://www.ncbi.nlm.nih.gov/pubmed/34274027
http://dx.doi.org/10.1186/s43058-021-00183-0
_version_ 1783723633956356096
author Lyon, Aaron R.
Coifman, Jessica
Cook, Heather
McRee, Erin
Liu, Freda F.
Ludwig, Kristy
Dorsey, Shannon
Koerner, Kelly
Munson, Sean A.
McCauley, Elizabeth
author_facet Lyon, Aaron R.
Coifman, Jessica
Cook, Heather
McRee, Erin
Liu, Freda F.
Ludwig, Kristy
Dorsey, Shannon
Koerner, Kelly
Munson, Sean A.
McCauley, Elizabeth
author_sort Lyon, Aaron R.
collection PubMed
description BACKGROUND: Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care. METHOD: The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability. RESULTS: Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. CONCLUSIONS: CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s43058-021-00183-0.
format Online
Article
Text
id pubmed-8285864
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher BioMed Central
record_format MEDLINE/PubMed
spelling pubmed-82858642021-07-19 The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability Lyon, Aaron R. Coifman, Jessica Cook, Heather McRee, Erin Liu, Freda F. Ludwig, Kristy Dorsey, Shannon Koerner, Kelly Munson, Sean A. McCauley, Elizabeth Implement Sci Commun Methodology BACKGROUND: Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care. METHOD: The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability. RESULTS: Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. CONCLUSIONS: CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s43058-021-00183-0. BioMed Central 2021-07-17 /pmc/articles/PMC8285864/ /pubmed/34274027 http://dx.doi.org/10.1186/s43058-021-00183-0 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
spellingShingle Methodology
Lyon, Aaron R.
Coifman, Jessica
Cook, Heather
McRee, Erin
Liu, Freda F.
Ludwig, Kristy
Dorsey, Shannon
Koerner, Kelly
Munson, Sean A.
McCauley, Elizabeth
The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
title The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
title_full The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
title_fullStr The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
title_full_unstemmed The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
title_short The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
title_sort cognitive walkthrough for implementation strategies (cwis): a pragmatic method for assessing implementation strategy usability
topic Methodology
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8285864/
https://www.ncbi.nlm.nih.gov/pubmed/34274027
http://dx.doi.org/10.1186/s43058-021-00183-0
work_keys_str_mv AT lyonaaronr thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT coifmanjessica thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT cookheather thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT mcreeerin thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT liufredaf thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT ludwigkristy thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT dorseyshannon thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT koernerkelly thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT munsonseana thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT mccauleyelizabeth thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT lyonaaronr cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT coifmanjessica cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT cookheather cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT mcreeerin cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT liufredaf cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT ludwigkristy cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT dorseyshannon cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT koernerkelly cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT munsonseana cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability
AT mccauleyelizabeth cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability