Instructive head-mounted display system: pointing device using a vision-based finger tracking technique applied to surgical education

INTRODUCTION: The head-mounted display (HMD) system is a novel personalized imaging monitoring system for use in a medical setting. AIM: To support interactive intraoperative communication among HMD wearers, we integrated vision-based finger tracking into our system as a pointing device. MATERIAL AND METHODS: Our vision-based finger tracking system is composed of a commercially available real-time video camera, which is mounted on the modern HMD, and computer software that enables tracking of the tip of the operator's index finger and superimposing the marker on the endoscopic view. RESULTS: We used this system in an experimental demonstration. The operator used the finger-tracking pointer to explain the intraoperative findings of transurethral resection for bladder cancer to medical students. CONCLUSIONS: This finger tracking system-based pointing device can function as a supportive tool for the HMD system, enabling interactive instruction and communication between the operator and other attending physicians or medical students.