Implementation and Validation of an Automated, Longitudinal Robotic Surgical Evaluation and Feedback Program at a High-volume Center and Impact on Training

Abstract

Background: Surgical education lacks a standardized, proficiency-based approach to evaluation and feedback. Objective: To assess the implementation and reception (ie, feasibility) of an automated, standardized, longitudinal surgical skill assessment and feedback system, and identify baseline trainee (resident and fellow) characteristics associated with achieving proficiency in robotic surgery while learning robotic-assisted laparoscopic prostatectomy. Design, setting, and participants: A quality improvement study assessing a pilot of a surgical experience tracking program was conducted over 1 yr. Participants were six fellows, eight residents, and nine attending surgeons at a tertiary cancer center. Intervention: Trainees underwent baseline self-assessment. After each surgery, an evaluation was completed independently by the trainee and attending surgeons. Performance was rated on a five-point anchored Likert scale (trainees were considered “proficient” when attending surgeons’ rating was ≥4). Technical skills were assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and Prostatectomy Assessment and Competency Evaluation (PACE). Outcome measurements and statistical analysis: Program success and utility were assessed by evaluating completion rates, evaluation completion times, and concordance rates between attending and trainee surgeons, and exit surveys. Baseline characteristics were assessed to determine associations with achieving proficiency. Results and limitations: Completion rates for trainees and attending surgeons were 72% and 77%, respectively. Fellows performed more steps/cases than residents (median [interquartile range]: 5 [3–7] and 3 [2–4], respectively; p < 0.01). Prior completion of robotics or laparoscopic skill courses and surgical experience measures were associated with achieving proficiency in multiple surgical steps and GEARS domains. Interclass correlation coefficients on individual components were 0.27–0.47 on GEARS domains. Conclusions: An automated surgical experience tracker with structured, longitudinal evaluation and feedback can be implemented with good participation and minimal participant time commitment, and can guide curricular development in a proficiency-based education program by identifying modifiable factors associated with proficiency, individualizing education, and identifying improvement areas within the education program. Patient summary: An automated, standardized, longitudinal surgical skill assessment and feedback system can be implemented successfully in surgical education settings and used to inform education plans and predict trainee proficiency.

Keywords

• Education
• Learning curve
• Robotics