Complete Object-Compositional Neural Implicit Surfaces With 3D Pseudo Supervision

Abstract

Neural implicit surface reconstruction has recently emerged as a prominent paradigm in multi-view 3D reconstruction using deep learning. In contrast to traditional multi-view stereo methods, signed distance function (SDF)-based approaches leverage neural networks to effectively represent 3D scenes. Furthermore, to reconstruct scenes and individual objects separately, some studies have extended the framework for object-compositional neural implicit surface reconstruction by utilizing 2D instance masks to supervise the SDF of each object. Nonetheless, these methods often reconstruct objects as partial shapes in scenes captured from sparse viewpoints or in complex scenes containing multiple objects. This issue primarily stems from the absence of a 3D prior, which fails to provide sufficient geometry for partially observed and occluded regions. We propose a framework for completing the partial object shapes of object-compositional neural implicit representation utilizing a diffusion-based 3D mesh generation model. The existing diffusion model, trained only on large-scale 3D object datasets, generates complete shapes from partial shapes; however, their results differ significantly from the objects in the scene. To complete the representation of partial shapes while ensuring shape consistency across multi-view images, we combine the SDF values, output by the diffusion model, with the object-compositional neural implicit representation. The combined representation is then volume-rendered to fine-tune the diffusion model utilizing a 2D prior. Furthermore, the complete shape generated by our method can serve as pseudo 3D priors to provide the geometry for the unobserved regions in object-compositional representation. Extensive experiments demonstrate that our novel framework significantly improves the reconstruction quality of unobserved regions.

Keywords

• Deep learning
• mesh generation
• surface reconstruction