Cargando…

Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial

IMPORTANCE: To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches. OBJECTIVE: To determine how VOA and remote expert instruc...

Descripción completa

Detalles Bibliográficos
Autores principales: Fazlollahi, Ali M., Bakhaidar, Mohamad, Alsayegh, Ahmad, Yilmaz, Recai, Winkler-Schwartz, Alexander, Mirchi, Nykan, Langleben, Ian, Ledwos, Nicole, Sabbagh, Abdulrahman J., Bajunaid, Khalid, Harley, Jason M., Del Maestro, Rolando F.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: American Medical Association 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8864513/
https://www.ncbi.nlm.nih.gov/pubmed/35191972
http://dx.doi.org/10.1001/jamanetworkopen.2021.49008
_version_ 1784655476306214912
author Fazlollahi, Ali M.
Bakhaidar, Mohamad
Alsayegh, Ahmad
Yilmaz, Recai
Winkler-Schwartz, Alexander
Mirchi, Nykan
Langleben, Ian
Ledwos, Nicole
Sabbagh, Abdulrahman J.
Bajunaid, Khalid
Harley, Jason M.
Del Maestro, Rolando F.
author_facet Fazlollahi, Ali M.
Bakhaidar, Mohamad
Alsayegh, Ahmad
Yilmaz, Recai
Winkler-Schwartz, Alexander
Mirchi, Nykan
Langleben, Ian
Ledwos, Nicole
Sabbagh, Abdulrahman J.
Bajunaid, Khalid
Harley, Jason M.
Del Maestro, Rolando F.
author_sort Fazlollahi, Ali M.
collection PubMed
description IMPORTANCE: To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches. OBJECTIVE: To determine how VOA and remote expert instruction compare in learners’ skill acquisition, affective, and cognitive outcomes during surgical simulation training. DESIGN, SETTING, AND PARTICIPANTS: This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021. INTERVENTIONS: The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback. MAIN OUTCOMES AND MEASURES: The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, −1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports. RESULTS: A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group. All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001). Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups. However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found. CONCLUSIONS AND RELEVANCE: In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT04700384
format Online
Article
Text
id pubmed-8864513
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher American Medical Association
record_format MEDLINE/PubMed
spelling pubmed-88645132022-03-04 Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial Fazlollahi, Ali M. Bakhaidar, Mohamad Alsayegh, Ahmad Yilmaz, Recai Winkler-Schwartz, Alexander Mirchi, Nykan Langleben, Ian Ledwos, Nicole Sabbagh, Abdulrahman J. Bajunaid, Khalid Harley, Jason M. Del Maestro, Rolando F. JAMA Netw Open Original Investigation IMPORTANCE: To better understand the emerging role of artificial intelligence (AI) in surgical training, efficacy of AI tutoring systems, such as the Virtual Operative Assistant (VOA), must be tested and compared with conventional approaches. OBJECTIVE: To determine how VOA and remote expert instruction compare in learners’ skill acquisition, affective, and cognitive outcomes during surgical simulation training. DESIGN, SETTING, AND PARTICIPANTS: This instructor-blinded randomized clinical trial included medical students (undergraduate years 0-2) from 4 institutions in Canada during a single simulation training at McGill Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal, Canada. Cross-sectional data were collected from January to April 2021. Analysis was conducted based on intention-to-treat. Data were analyzed from April to June 2021. INTERVENTIONS: The interventions included 5 feedback sessions, 5 minutes each, during a single 75-minute training, including 5 practice sessions followed by 1 realistic virtual reality brain tumor resection. The 3 intervention arms included 2 treatment groups, AI audiovisual metric-based feedback (VOA group) and synchronous verbal scripted debriefing and instruction from a remote expert (instructor group), and a control group that received no feedback. MAIN OUTCOMES AND MEASURES: The coprimary outcomes were change in procedural performance, quantified as Expertise Score by a validated assessment algorithm (Intelligent Continuous Expertise Monitoring System [ICEMS]; range, −1.00 to 1.00) for each practice resection, and learning and retention, measured from performance in realistic resections by ICEMS and blinded Objective Structured Assessment of Technical Skills (OSATS; range 1-7). Secondary outcomes included strength of emotions before, during, and after the intervention and cognitive load after intervention, measured in self-reports. RESULTS: A total of 70 medical students (41 [59%] women and 29 [41%] men; mean [SD] age, 21.8 [2.3] years) from 4 institutions were randomized, including 23 students in the VOA group, 24 students in the instructor group, and 23 students in the control group. All participants were included in the final analysis. ICEMS assessed 350 practice resections, and ICEMS and OSATS evaluated 70 realistic resections. VOA significantly improved practice Expertise Scores by 0.66 (95% CI, 0.55 to 0.77) points compared with the instructor group and by 0.65 (95% CI, 0.54 to 0.77) points compared with the control group (P < .001). Realistic Expertise Scores were significantly higher for the VOA group compared with instructor (mean difference, 0.53 [95% CI, 0.40 to 0.67] points; P < .001) and control (mean difference. 0.49 [95% CI, 0.34 to 0.61] points; P < .001) groups. Mean global OSATS ratings were not statistically significant among the VOA (4.63 [95% CI, 4.06 to 5.20] points), instructor (4.40 [95% CI, 3.88-4.91] points), and control (3.86 [95% CI, 3.44 to 4.27] points) groups. However, on the OSATS subscores, VOA significantly enhanced the mean OSATS overall subscore compared with the control group (mean difference, 1.04 [95% CI, 0.13 to 1.96] points; P = .02), whereas expert instruction significantly improved OSATS subscores for instrument handling vs control (mean difference, 1.18 [95% CI, 0.22 to 2.14]; P = .01). No significant differences in cognitive load, positive activating, and negative emotions were found. CONCLUSIONS AND RELEVANCE: In this randomized clinical trial, VOA feedback demonstrated superior performance outcome and skill transfer, with equivalent OSATS ratings and cognitive and emotional responses compared with remote expert instruction, indicating advantages for its use in simulation training. TRIAL REGISTRATION: ClinicalTrials.gov Identifier: NCT04700384 American Medical Association 2022-02-22 /pmc/articles/PMC8864513/ /pubmed/35191972 http://dx.doi.org/10.1001/jamanetworkopen.2021.49008 Text en Copyright 2022 Fazlollahi AM et al. JAMA Network Open. https://creativecommons.org/licenses/by/4.0/This is an open access article distributed under the terms of the CC-BY License.
spellingShingle Original Investigation
Fazlollahi, Ali M.
Bakhaidar, Mohamad
Alsayegh, Ahmad
Yilmaz, Recai
Winkler-Schwartz, Alexander
Mirchi, Nykan
Langleben, Ian
Ledwos, Nicole
Sabbagh, Abdulrahman J.
Bajunaid, Khalid
Harley, Jason M.
Del Maestro, Rolando F.
Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial
title Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial
title_full Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial
title_fullStr Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial
title_full_unstemmed Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial
title_short Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial
title_sort effect of artificial intelligence tutoring vs expert instruction on learning simulated surgical skills among medical students: a randomized clinical trial
topic Original Investigation
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8864513/
https://www.ncbi.nlm.nih.gov/pubmed/35191972
http://dx.doi.org/10.1001/jamanetworkopen.2021.49008
work_keys_str_mv AT fazlollahialim effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT bakhaidarmohamad effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT alsayeghahmad effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT yilmazrecai effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT winklerschwartzalexander effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT mirchinykan effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT langlebenian effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT ledwosnicole effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT sabbaghabdulrahmanj effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT bajunaidkhalid effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT harleyjasonm effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial
AT delmaestrorolandof effectofartificialintelligencetutoringvsexpertinstructiononlearningsimulatedsurgicalskillsamongmedicalstudentsarandomizedclinicaltrial