Cargando…

The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria

BACKGROUND: The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be dif...

Descripción completa

Detalles Bibliográficos
Autores principales: Floyd, Erin, Adler, Shelley R., Crane, Rebecca S., Brewer, Judson, Moran, Patricia, Richler, Robert, Hartogensis, Wendy, Kuyken, Willem, Hecht, Frederick M.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: SAGE Publications 2023
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10196544/
https://www.ncbi.nlm.nih.gov/pubmed/37216036
http://dx.doi.org/10.1177/27536130221149966
_version_ 1785044379059093504
author Floyd, Erin
Adler, Shelley R.
Crane, Rebecca S.
Brewer, Judson
Moran, Patricia
Richler, Robert
Hartogensis, Wendy
Kuyken, Willem
Hecht, Frederick M.
author_facet Floyd, Erin
Adler, Shelley R.
Crane, Rebecca S.
Brewer, Judson
Moran, Patricia
Richler, Robert
Hartogensis, Wendy
Kuyken, Willem
Hecht, Frederick M.
author_sort Floyd, Erin
collection PubMed
description BACKGROUND: The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be difficult to obtain, share for assessments, and which increase privacy concerns for participants. Audio-only recordings might be a useful alternative, but their reliability is unknown. OBJECTIVE: To assess evaluator perception of the rating process and inter-rater reliability of MBI:TAC ratings using audio-only recordings. METHODS: We prepared audio-only files from video recordings of 21 previously rated Mindfulness-Based Stress Reduction teachers. Each audio recording was rated by 3 trained MBI:TAC assessors drawn from a pool of 12 who had previously participated in rating the video recordings. Teachers were rated by evaluators who had not viewed the video recording and did not know the teacher. We then conducted semi-structured interviews with evaluators. RESULTS: On the 6 MBI:TAC domains, the intraclass correlation coefficients (ICCs) for audio recordings ranged from .53 to .69 using an average across 3 evaluators. Using a single rating resulted in lower ICCs (.27-.38). Bland-Altman plots showed audio ratings had little consistent bias compared to video recordings and agreed more closely for teachers with higher ratings. Qualitative analysis identified 3 themes: video recordings were particularly helpful when rating less skillful teachers, video recordings tended to provide a more complete picture for rating, and audio rating had some positive features. CONCLUSIONS: Inter-rater reliability of the MBI:TAC using audio-only recordings was adequate for many research and clinical purposes, and reliability is improved when using an average across several evaluators. Ratings using audio-only recordings may be more challenging when rating less experienced teachers.
format Online
Article
Text
id pubmed-10196544
institution National Center for Biotechnology Information
language English
publishDate 2023
publisher SAGE Publications
record_format MEDLINE/PubMed
spelling pubmed-101965442023-05-20 The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria Floyd, Erin Adler, Shelley R. Crane, Rebecca S. Brewer, Judson Moran, Patricia Richler, Robert Hartogensis, Wendy Kuyken, Willem Hecht, Frederick M. Glob Adv Integr Med Health Intervention Fidelity in Mindfulness-Based Research and Practice BACKGROUND: The Mindfulness-Based Interventions: Teaching Assessment Criteria (MBI:TAC) is an important tool for assessing teacher skill and aspects of the fidelity of mindfulness-based interventions, but prior research on and implementation of the MBI:TAC has used video recordings, which can be difficult to obtain, share for assessments, and which increase privacy concerns for participants. Audio-only recordings might be a useful alternative, but their reliability is unknown. OBJECTIVE: To assess evaluator perception of the rating process and inter-rater reliability of MBI:TAC ratings using audio-only recordings. METHODS: We prepared audio-only files from video recordings of 21 previously rated Mindfulness-Based Stress Reduction teachers. Each audio recording was rated by 3 trained MBI:TAC assessors drawn from a pool of 12 who had previously participated in rating the video recordings. Teachers were rated by evaluators who had not viewed the video recording and did not know the teacher. We then conducted semi-structured interviews with evaluators. RESULTS: On the 6 MBI:TAC domains, the intraclass correlation coefficients (ICCs) for audio recordings ranged from .53 to .69 using an average across 3 evaluators. Using a single rating resulted in lower ICCs (.27-.38). Bland-Altman plots showed audio ratings had little consistent bias compared to video recordings and agreed more closely for teachers with higher ratings. Qualitative analysis identified 3 themes: video recordings were particularly helpful when rating less skillful teachers, video recordings tended to provide a more complete picture for rating, and audio rating had some positive features. CONCLUSIONS: Inter-rater reliability of the MBI:TAC using audio-only recordings was adequate for many research and clinical purposes, and reliability is improved when using an average across several evaluators. Ratings using audio-only recordings may be more challenging when rating less experienced teachers. SAGE Publications 2023-05-17 /pmc/articles/PMC10196544/ /pubmed/37216036 http://dx.doi.org/10.1177/27536130221149966 Text en © The Author(s) 2023 https://creativecommons.org/licenses/by-nc/4.0/This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License (https://creativecommons.org/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access pages (https://us.sagepub.com/en-us/nam/open-access-at-sage).
spellingShingle Intervention Fidelity in Mindfulness-Based Research and Practice
Floyd, Erin
Adler, Shelley R.
Crane, Rebecca S.
Brewer, Judson
Moran, Patricia
Richler, Robert
Hartogensis, Wendy
Kuyken, Willem
Hecht, Frederick M.
The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria
title The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria
title_full The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria
title_fullStr The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria
title_full_unstemmed The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria
title_short The Reliability of Rating via Audio-Recording Using the Mindfulness-Based Interventions: Teaching Assessment Criteria
title_sort reliability of rating via audio-recording using the mindfulness-based interventions: teaching assessment criteria
topic Intervention Fidelity in Mindfulness-Based Research and Practice
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10196544/
https://www.ncbi.nlm.nih.gov/pubmed/37216036
http://dx.doi.org/10.1177/27536130221149966
work_keys_str_mv AT floyderin thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT adlershelleyr thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT cranerebeccas thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT brewerjudson thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT moranpatricia thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT richlerrobert thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT hartogensiswendy thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT kuykenwillem thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT hechtfrederickm thereliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT floyderin reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT adlershelleyr reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT cranerebeccas reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT brewerjudson reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT moranpatricia reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT richlerrobert reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT hartogensiswendy reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT kuykenwillem reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria
AT hechtfrederickm reliabilityofratingviaaudiorecordingusingthemindfulnessbasedinterventionsteachingassessmentcriteria