Complex-Motion NeRF: Joint Reconstruction and Pose Optimization With Motion and Depth Priors

Abstract

We present Complex-Motion Neural Radiance Fields (CM-NeRF), which is a method that leverages motion and depth priors to optimize neural 3D scene representations and complex 6-DoF camera motions jointly. Although NeRF has achieved remarkable success in neural rendering and reconstruction, they require accurate camera poses and sufficient input for realizing high-quality novel view synthesis. We aim to recover accurate camera motion and NeRF simultaneously by effectively using motion and depth priors when a few input images are available. Moreover, our approach enables stable pose estimation and efficient view recovery in challenging and complex camera movements in addition to forward-facing camera motions. Considering the confidence of the depth, we use the depth map to guide the ray sampling and leverage depth information to accelerate the NeRF network training. Our experiments demonstrate the effectiveness of the CM-NeRF method in real-world scenarios involving challenging and complex camera motions. These results are non-trivial and may present significant variations compared to state-of-the-art techniques. The CM-NeRF demonstrates stable camera-pose estimation and efficient view recovery with only five training views of real-world data.

Keywords

• Neural radiance fields
• neural rendering
• novel view synthesis