Neural surface reconstruction with saliency‐guided sampling in multi‐view

Abstract

Abstract In this work, a neural surface reconstruction framework is presented. In order to perform neural surface reconstruction using 2D supervision, a weighted random sampling based on saliency is introduced for training the deep neural network. In the proposed method, self‐attention is used to detect the saliency of input 2D images. The saliency map, that is, the weight matrix of the weighted random sampling, is used to sample the training samples. As a result, more samples in the reconstructed object area are collected. Moreover, an update strategy for weight based on sampling frequency is adopted to avoid the points that cannot be sampled all the time. The experiments are implemented in real‐world 2D images of objects with different material properties and lighting conditions based on the DTU dataset. The results show that the proposed method produces more detailed 3D surfaces, and the rendered results are closer to the raw images visually. In addition, the mean of peak signal‐to‐noise ratio (PNSR) is also improved.

Keywords

• deep learning
• neural surface reconstruction
• self‐attention
• weighted random sampling