Development of an Augmented Reality Surgical Trainer for Minimally Invasive Pancreatic Surgery

Abstract

Robot-assisted minimally invasive surgery offers advantages over traditional laparoscopic surgery, including precision and improved patient outcomes. However, its complexity requires extensive training, leading to the development of simulators that still face challenges such as limited feedback and lack of realism. This study presents an augmented reality-based surgical simulator tailored for minimally invasive pancreatic surgery, integrating an innovative parallel robot, real-time AI-driven force estimation, and haptic feedback. Using Unity and the HoloLens 2, the simulator offers a realistic augmented environment, enhancing spatial awareness and planning in surgical scenarios. A convolutional neural network (CNN) model predicts forces without physical sensors, achieving a mean absolute error of 0.0244 N. Tests indicate a strong correlation between applied and predicted forces, with a haptic feedback latency of 65 ms, suitable for real-time applications. Its modularity makes the simulator accessible for training and preoperative planning, addressing gaps in current robotic surgery training tools while setting the stage for future improvements and broader integration.

Keywords

• surgical simulator
• augmented reality
• parallel robot
• artificial intelligence
• minimally invasive surgery
• robotic-assisted surgery