Cargando…

Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training

BACKGROUND: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency deve...

Descripción completa

Detalles Bibliográficos
Autores principales: Siau, Keith, Crossley, James, Dunckley, Paul, Johnson, Gavin, Feeney, Mark, Hawkes, Neil D., Beales, Ian L. P.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Springer US 2019
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6946748/
https://www.ncbi.nlm.nih.gov/pubmed/30911922
http://dx.doi.org/10.1007/s00464-019-06737-7
_version_ 1783485428010057728
author Siau, Keith
Crossley, James
Dunckley, Paul
Johnson, Gavin
Feeney, Mark
Hawkes, Neil D.
Beales, Ian L. P.
author_facet Siau, Keith
Crossley, James
Dunckley, Paul
Johnson, Gavin
Feeney, Mark
Hawkes, Neil D.
Beales, Ian L. P.
author_sort Siau, Keith
collection PubMed
description BACKGROUND: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. METHODS: This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. RESULTS: In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. CONCLUSION: This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s00464-019-06737-7) contains supplementary material, which is available to authorised users.
format Online
Article
Text
id pubmed-6946748
institution National Center for Biotechnology Information
language English
publishDate 2019
publisher Springer US
record_format MEDLINE/PubMed
spelling pubmed-69467482020-01-21 Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training Siau, Keith Crossley, James Dunckley, Paul Johnson, Gavin Feeney, Mark Hawkes, Neil D. Beales, Ian L. P. Surg Endosc Article BACKGROUND: Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of the formative direct observation of procedural skills (DOPS) assessment tool in diagnostic gastroscopy and study competency development using DOPS. METHODS: This was a prospective multicentre (N = 275) analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. RESULTS: In total, 10086 DOPS were submitted for 987 trainees. Exploratory factor analysis identified three distinct item groupings, representing ‘pre-procedure’, ‘technical’, and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of three assessments from three assessors was sufficient to achieve the reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of ‘pre-procedure’ (100–124 procedures), ‘technical’ (150–174 procedures), ‘post-procedure non-technical’ skills (200–224 procedures), and global competency (225–249 procedures). Higher lifetime procedure count, DOPS count, surgical trainees and assessors, higher trainee seniority, and lower case difficulty were significant multivariable predictors of DOPS competence. CONCLUSION: This study establishes milestones for competency acquisition during gastroscopy training and provides validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s00464-019-06737-7) contains supplementary material, which is available to authorised users. Springer US 2019-03-25 2020 /pmc/articles/PMC6946748/ /pubmed/30911922 http://dx.doi.org/10.1007/s00464-019-06737-7 Text en © The Author(s) 2019 Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
spellingShingle Article
Siau, Keith
Crossley, James
Dunckley, Paul
Johnson, Gavin
Feeney, Mark
Hawkes, Neil D.
Beales, Ian L. P.
Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
title Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
title_full Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
title_fullStr Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
title_full_unstemmed Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
title_short Direct observation of procedural skills (DOPS) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
title_sort direct observation of procedural skills (dops) assessment in diagnostic gastroscopy: nationwide evidence of validity and competency development during training
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6946748/
https://www.ncbi.nlm.nih.gov/pubmed/30911922
http://dx.doi.org/10.1007/s00464-019-06737-7
work_keys_str_mv AT siaukeith directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT crossleyjames directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT dunckleypaul directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT johnsongavin directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT feeneymark directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT hawkesneild directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT bealesianlp directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining
AT directobservationofproceduralskillsdopsassessmentindiagnosticgastroscopynationwideevidenceofvalidityandcompetencydevelopmentduringtraining