Details preserved unsupervised depth estimation by fusing traditional stereo knowledge from laparoscopic images

Abstract

Depth estimation plays an important role in vision-based laparoscope surgical navigation systems. Most learning-based depth estimation methods require ground truth depth or disparity images for training; however, these data are difficult to obtain in laparoscopy. The authors present an unsupervised learning depth estimation approach by fusing traditional stereo knowledge. The traditional stereo method is used to generate proxy disparity labels, in which unreliable depth measurements are removed via a confidence measure to improve stereo accuracy. The disparity images are generated by training a dual encoder–decoder convolutional neural network from rectified stereo images coupled with proxy labels generated by the traditional stereo method. A principled mask is computed to exclude the pixels, which are not seen in one of views due to parallax effects from the calculation of loss function. Moreover, the neighbourhood smoothness term is employed to constrain neighbouring pixels with similar appearances to generate a smooth depth surface. This approach can make the depth of the projected point cloud closer to the real surgical site and preserve realistic details. The authors demonstrate the performance of the method by training and evaluation with a partial nephrectomy da Vinci surgery dataset and heart phantom data from the Hamlyn Centre.

Keywords

• stereo image processing
• surgery
• image reconstruction
• unsupervised learning
• medical image processing
• phantoms
• image motion analysis
• convolutional neural nets
• traditional stereo method
• proxy disparity labels
• unreliable depth measurements
• confidence measure
• stereo accuracy
• disparity images
• rectified stereo images
• proxy labels
• smooth depth surface
• unsupervised depth estimation
• traditional stereo knowledge
• laparoscopic images
• vision-based laparoscope surgical navigation systems
• truth depth
• unsupervised learning depth estimation approach
• dual encoder-decoder convolutional neural network
• loss function
• principled mask
• parallax effects
• neighbourhood smoothness term
• constrain neighbouring pixels
• partial nephrectomy da vinci surgery dataset
• heart phantom data
• hamlyn centre