Use of Machine Learning to Assess Cataract Surgery Skill Level With Tool Detection

PURPOSE: To develop a method for objective analysis of the reproducible steps in routine cataract surgery. DESIGN: Prospective study; machine learning. PARTICIPANTS: Deidentified faculty and trainee surgical videos. METHODS: Consecutive cataract surgeries performed by a faculty or trainee surgeon in an ophthalmology residency program over 6 months were collected and labeled according to degrees of difficulty. An existing image classification network, ResNet 152, was fine-tuned for tool detection in cataract surgery to allow for automatic identification of each unique surgical instrument. Individual microscope video frame windows were subsequently encoded as a vector. The relation between vector encodings and perceived skill using k-fold user-out cross-validation was examined. Algorithms were evaluated using area under the receiver operating characteristic curve (AUC) and the classification accuracy. MAIN OUTCOME MEASURES: Accuracy of tool detection and skill assessment. RESULTS: In total, 391 consecutive cataract procedures with 209 routine cases were used. Our model achieved an AUC ranging from 0.933 to 0.998 for tool detection. For skill classification, AUC was 0.550 (95% confidence interval [CI], 0.547–0.553) with an accuracy of 54.3% (95% CI, 53.9%–54.7%) for a single snippet, AUC was 0.570 (0.565–0.575) with an accuracy of 57.8% (56.8%–58.7%) for a single surgery, and AUC was 0.692 (0.659–0.758) with an accuracy of 63.3% (56.8%–69.8%) for a single user given all their trials. CONCLUSIONS: Our research shows that machine learning can accurately and independently identify distinct cataract surgery tools in videos, which is crucial for comparing the use of the tool in a step. However, it is more challenging for machine learning to accurately differentiate overall and specific step skill to assess the level of training or expertise. FINANCIAL DISCLOSURE(S): The author(s) have no proprietary or commercial interest in any materials discussed in this article.