Reconstruction of Photorealistic 3D Urban Scenes Using Radiance Fields as Digital Twins for Autonomous Driving

Abstract

We explore the use of radiance fields (RFs) to reconstruct photorealistic 3D urban scenes, creating digital twins (DTs) for autonomous driving (AD) by leveraging Nerfacto and Splatfacto models integrated with the CARLA simulator. Our research demonstrates that publicly available RFs can be utilized through Nerfstudio library to create photorealistic urban scenes and extract arbitrary images based on the camera pose. These scenes can serve as simulations for AD or as DT repositories for static environments within the vehicular metaverse. Additionally, we quantitatively evaluate RF models and use masking to remove dynamic objects, successfully simulating real-world scenarios. Quantitative evaluation shows that the Splatfacto model achieves a peak signal-to-noise ratio (PSNR) of up to 26.40, a structural similarity index measure (SSIM) of 0.84, and a learned perceptual image patch similarity (LPIPS) score of 0.21, consistently outperforming the Nerfacto model.

Keywords

• autonomous driving
• carla simulator
• digital twin
• radiance fields
• vehicular metaverse