Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BACKGROUND: Videos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment. METHODS: A multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird’s-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment. RESULTS AND DISCUSSION: The scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio–video quality, and questionable feasibility for assessing readiness for practice. CONCLUSION: Video-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio–video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s13018-023-03557-3.