Using a novel virtual-reality simulator to assess performance in lumbar puncture: a validation study

BACKGROUND: A lumbar puncture procedure’s success depends on a competent physician minimizing the risk of failing to get a sample and avoiding complications such as post-dural headache. A new virtual-reality simulator might be helpful in deciding when a physician is competent to perform lumbar puncture. We aimed to investigate validity evidence for a simulator-based test in lumbar puncture and establish a pass/fail standard to allow a mastery learning training program. METHODS: Validity evidence was investigated using Messick’s framework by including participants who were novices, intermediates, or experienced in lumbar puncture. Each participant performed two lumbar puncture procedures on the simulator, and fifty-nine predefined simulator metrics were automatically recorded. Cronbach’s alpha was used to explore internal consistency reliability. Intergroup comparisons were made using independent sample t-tests with Tukey’s correction for multiple comparisons. The learning effect was explored using paired sample t-test analysis, and a pass/fail standard was established using the contrasting groups’ method. RESULTS: 73 novices, 18 intermediates, and 19 physicians performed the test resulting in a total of 220 procedures. 25 metrics (42.4%) had good discriminatory ability, and the reliability of these metrics was good, Cronbach’s α = 0.81. The experienced physicians were significantly better than the novices (18.3 vs. 13.3, p < 0.001), and the pass/fail standard was established at 16 points. This standard resulted in 22 (30.1%) novices passing (i.e., false positives) and 5 (26.3%) physicians failing (i.e., false negatives). CONCLUSION: This study provides validity evidence for a simulator-based test of lumbar puncture competence. The test can help ensure basic competence at the end of a simulation-based training program for trainees, i.e., a mastery learning training program. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12909-023-04806-z.