Validation studies of virtual reality simulation performance metrics for mechanical thrombectomy in ischemic stroke

INTRODUCTION: Mechanical thrombectomy (MT) has transformed the treatment of ischemic stroke. However, patient access to MT may be limited due to a shortage of doctors specifically trained to perform MT. The studies reported here were done to (1) develop, operationally define, and seek consensus from procedure experts on the metrics which best characterize a reference procedure for the performance of an MT for ischemic stroke and (2) evaluate their construct validity when implemented in a virtual reality (VR) simulation. METHODS: In study 1, the metrics for a reference approach to an MT procedure for ischemic stroke of 10 phases, 46 steps, and 56 errors and critical errors, were presented to an international Delphi panel of 21 consultant level interventional neuroradiologists (INRs). In study 2, the metrics were used to assess 8 expert and 10 novice INRs performing a VR simulated routine MT procedure. RESULTS: In study 1, the Delphi panel reached consensus on the appropriateness of the procedure metrics for a reference approach to MT in ischemic stroke. Group differences in median scores in study 2 demonstrated that experienced INRs performed the case 19% faster (P=0.029), completed 40% more procedure phases (P=0.009), 20% more steps (P=0.012), and made 42% fewer errors (P=0.016) than the novice group. CONCLUSIONS: The international Delphi panel agreed metrics implemented in a VR simulation of MT distinguished between the computer scored procedure performance of INR experts and novices. The studies reported here support the demonstration of face, content, and construct validity of the MT metrics.