Weakly supervised segmentation for real-time surgical tool tracking

Abstract

Surgical tool tracking has a variety of applications in different surgical scenarios. Electromagnetic (EM) tracking can be utilised for tool tracking, but the accuracy is often limited by magnetic interference. Vision-based methods have also been suggested; however, tracking robustness is limited by specular reflection, occlusions, and blurriness observed in the endoscopic image. Recently, deep learning-based methods have shown competitive performance on segmentation and tracking of surgical tools. The main bottleneck of these methods lies in acquiring a sufficient amount of pixel-wise, annotated training data, which demands substantial labour costs. To tackle this issue, the authors propose a weakly supervised method for surgical tool segmentation and tracking based on hybrid sensor systems. They first generate semantic labellings using EM tracking and laparoscopic image processing concurrently. They then train a light-weight deep segmentation network to obtain a binary segmentation mask that enables tool tracking. To the authors’ knowledge, the proposed method is the first to integrate EM tracking and laparoscopic image processing for generation of training labels. They demonstrate that their framework achieves accurate, automatic tool segmentation (i.e. without any manual labelling of the surgical tool to be tracked) and robust tool tracking in laparoscopic image sequences.

Keywords

• endoscopes
• computer vision
• learning (artificial intelligence)
• surgery
• image segmentation
• medical image processing
• medical robotics
• image sequences
• tracking
• neural nets
• annotated training data
• surgical tool segmentation
• laparoscopic image processing
• light-weight deep segmentation network
• binary segmentation mask
• automatic tool segmentation
• robust tool tracking
• supervised segmentation
• real-time surgical tool tracking
• surgical scenarios
• electromagnetic tracking
• vision-based methods
• tracking robustness
• deep learning-based methods
• pixel-wise training data