Fusing feature consistency across views for multi-view stereo

Abstract

Abstract Dense feature matching is of significant importance in the learning-based multi-view stereo (MVS) pipeline. Feature consistency across views is a key factor affecting feature matching and plays a crucial role in the reconstruction of objects by the learning-based MVS method. This factor has not been adequately considered in previous studies. To address this issue, we first introduce the color invariance module, which derives a set of object color properties that are independent of illumination and viewpoint from the physics-based reflection model and the RGB-based Gaussian color model. This module highlights consistent feature representations across views. Meanwhile, it also facilitates the learning of feature consistency as prior knowledge. We then propose two pixel-wise feature losses. These losses further encourage and supervise the image feature extractor to learn consistent features for pixels with the same meaning in multiple views. By focusing on feature consistency across views, we enable the network to perceive similar visual representations among multiple views and boost the performance of the MVS task. To demonstrate the rationality and effectiveness of these strategies for the learning-based MVS, we conduct experiments on the DTU and Tanks & Temples datasets, achieving better reconstruction completeness. Compared to other state-of-the-art methods, our method also shows better generalization ability on the ETH3D dataset.

Keywords

• Deep learning
• Multi-view stereo
• Depth map
• Feature consistency