Ophthalmology Science (Mar 2023)

Use of Machine Learning to Assess Cataract Surgery Skill Level With Tool Detection

  • Jessica Ruzicki, MD, FRCSC,
  • Matthew Holden, PhD,
  • Stephanie Cheon, MD,
  • Tamas Ungi, MD, PhD,
  • Rylan Egan, PhD,
  • Christine Law, MD, FRCSC

Journal volume & issue
Vol. 3, no. 1
p. 100235

Abstract

Read online

Purpose: To develop a method for objective analysis of the reproducible steps in routine cataract surgery. Design: Prospective study; machine learning. Participants: Deidentified faculty and trainee surgical videos. Methods: Consecutive cataract surgeries performed by a faculty or trainee surgeon in an ophthalmology residency program over 6 months were collected and labeled according to degrees of difficulty. An existing image classification network, ResNet 152, was fine-tuned for tool detection in cataract surgery to allow for automatic identification of each unique surgical instrument. Individual microscope video frame windows were subsequently encoded as a vector. The relation between vector encodings and perceived skill using k-fold user-out cross-validation was examined. Algorithms were evaluated using area under the receiver operating characteristic curve (AUC) and the classification accuracy. Main Outcome Measures: Accuracy of tool detection and skill assessment. Results: In total, 391 consecutive cataract procedures with 209 routine cases were used. Our model achieved an AUC ranging from 0.933 to 0.998 for tool detection. For skill classification, AUC was 0.550 (95% confidence interval [CI], 0.547–0.553) with an accuracy of 54.3% (95% CI, 53.9%–54.7%) for a single snippet, AUC was 0.570 (0.565–0.575) with an accuracy of 57.8% (56.8%–58.7%) for a single surgery, and AUC was 0.692 (0.659–0.758) with an accuracy of 63.3% (56.8%–69.8%) for a single user given all their trials. Conclusions: Our research shows that machine learning can accurately and independently identify distinct cataract surgery tools in videos, which is crucial for comparing the use of the tool in a step. However, it is more challenging for machine learning to accurately differentiate overall and specific step skill to assess the level of training or expertise. Financial Disclosure(s): The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Keywords