Cargando…
The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability
BACKGROUND: Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field l...
Autores principales: | , , , , , , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
BioMed Central
2021
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8285864/ https://www.ncbi.nlm.nih.gov/pubmed/34274027 http://dx.doi.org/10.1186/s43058-021-00183-0 |
_version_ | 1783723633956356096 |
---|---|
author | Lyon, Aaron R. Coifman, Jessica Cook, Heather McRee, Erin Liu, Freda F. Ludwig, Kristy Dorsey, Shannon Koerner, Kelly Munson, Sean A. McCauley, Elizabeth |
author_facet | Lyon, Aaron R. Coifman, Jessica Cook, Heather McRee, Erin Liu, Freda F. Ludwig, Kristy Dorsey, Shannon Koerner, Kelly Munson, Sean A. McCauley, Elizabeth |
author_sort | Lyon, Aaron R. |
collection | PubMed |
description | BACKGROUND: Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care. METHOD: The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability. RESULTS: Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. CONCLUSIONS: CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s43058-021-00183-0. |
format | Online Article Text |
id | pubmed-8285864 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2021 |
publisher | BioMed Central |
record_format | MEDLINE/PubMed |
spelling | pubmed-82858642021-07-19 The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability Lyon, Aaron R. Coifman, Jessica Cook, Heather McRee, Erin Liu, Freda F. Ludwig, Kristy Dorsey, Shannon Koerner, Kelly Munson, Sean A. McCauley, Elizabeth Implement Sci Commun Methodology BACKGROUND: Implementation strategies have flourished in an effort to increase integration of research evidence into clinical practice. Most strategies are complex, socially mediated processes. Many are complicated, expensive, and ultimately impractical to deliver in real-world settings. The field lacks methods to assess the extent to which strategies are usable and aligned with the needs and constraints of the individuals and contexts who will deliver or receive them. Drawn from the field of human-centered design, cognitive walkthroughs are an efficient assessment method with potential to identify aspects of strategies that may inhibit their usability and, ultimately, effectiveness. This article presents a novel walkthrough methodology for evaluating strategy usability as well as an example application to a post-training consultation strategy to support school mental health clinicians to adopt measurement-based care. METHOD: The Cognitive Walkthrough for Implementation Strategies (CWIS) is a pragmatic, mixed-methods approach for evaluating complex, socially mediated implementation strategies. CWIS includes six steps: (1) determine preconditions; (2) hierarchical task analysis; (3) task prioritization; (4) convert tasks to scenarios; (5) pragmatic group testing; and (6) usability issue identification, classification, and prioritization. A facilitator conducted two group testing sessions with clinician users (N = 10), guiding participants through 6 scenarios and 11 associated subtasks. Clinicians reported their anticipated likelihood of completing each subtask and provided qualitative justifications during group discussion. Following the walkthrough sessions, users completed an adapted quantitative assessment of strategy usability. RESULTS: Average anticipated success ratings indicated substantial variability across participants and subtasks. Usability ratings (scale 0–100) of the consultation protocol averaged 71.3 (SD = 10.6). Twenty-one usability problems were identified via qualitative content analysis with consensus coding, and classified by severity and problem type. High-severity problems included potential misalignment between consultation and clinical service timelines as well as digressions during consultation processes. CONCLUSIONS: CWIS quantitative usability ratings indicated that the consultation protocol was at the low end of the “acceptable” range (based on norms from the unadapted scale). Collectively, the 21 resulting usability issues explained the quantitative usability data and provided specific direction for usability enhancements. The current study provides preliminary evidence for the utility of CWIS to assess strategy usability and generate a blueprint for redesign. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s43058-021-00183-0. BioMed Central 2021-07-17 /pmc/articles/PMC8285864/ /pubmed/34274027 http://dx.doi.org/10.1186/s43058-021-00183-0 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) . The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/ (https://creativecommons.org/publicdomain/zero/1.0/) ) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
spellingShingle | Methodology Lyon, Aaron R. Coifman, Jessica Cook, Heather McRee, Erin Liu, Freda F. Ludwig, Kristy Dorsey, Shannon Koerner, Kelly Munson, Sean A. McCauley, Elizabeth The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability |
title | The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability |
title_full | The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability |
title_fullStr | The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability |
title_full_unstemmed | The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability |
title_short | The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability |
title_sort | cognitive walkthrough for implementation strategies (cwis): a pragmatic method for assessing implementation strategy usability |
topic | Methodology |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8285864/ https://www.ncbi.nlm.nih.gov/pubmed/34274027 http://dx.doi.org/10.1186/s43058-021-00183-0 |
work_keys_str_mv | AT lyonaaronr thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT coifmanjessica thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT cookheather thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT mcreeerin thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT liufredaf thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT ludwigkristy thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT dorseyshannon thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT koernerkelly thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT munsonseana thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT mccauleyelizabeth thecognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT lyonaaronr cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT coifmanjessica cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT cookheather cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT mcreeerin cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT liufredaf cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT ludwigkristy cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT dorseyshannon cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT koernerkelly cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT munsonseana cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability AT mccauleyelizabeth cognitivewalkthroughforimplementationstrategiescwisapragmaticmethodforassessingimplementationstrategyusability |