Machine Learning Approaches for Evaluating the Progress of Surgical Training on a Virtual Reality Simulator

Abstract

Automated assessment of surgical skills is crucial for the successful training of junior surgeons. Twenty-three medical students followed a structured training curriculum on a laparoscopic virtual reality (VR) simulator. Three surgical tasks with significant educational merit were considered (Tasks 5, 6, and 7). We evaluated seven machine learning (ML) models for classifying the students’ trials into two and three classes based on the progress of training (Beginning vs. End and Beginning vs. Middle vs. End). Additionally, we evaluated the same ML framework and a deep learning approach (LSTM) for predicting the remaining number of trials required to complete the training proficiently. A model-agnostic technique from the domain of explainable artificial intelligence (XAI) was also utilized to obtain interpretations of the employed black-box ML classifiers. For 2-class classification, the best model showed an accuracy of 97.1%, 96.9%, and 75.7% for Task 5, 6, and 7, respectively, whereas for 3-class classification, the corresponding accuracy was 96.3%, 95.9%, and 99.7%, respectively. The best regression algorithm was LSTM with a Mean Absolute Error of 4 (Task 5) and 3.6 trials (Tasks 6, 7). According to XAI, the kinematic parameters have a stronger impact on the classification decision than the goal-oriented metrics.

Keywords

• minimally invasive surgery
• machine learning
• VR simulator
• skills assessment