PIXGAN-Drone: 3D Avatar of Human Body Reconstruction From Multi-View 2D Images

Abstract

Study is being conducted on training Generative Adversarial Networks (GANs) from 2D datasets to generate 3D human body avatars. Numerous applications, such as virtual reality, sports analysis, cinematography, surveillance, and more, have advanced significantly as a result of the promising research in this subject. Aerial photography sensors together with drone active tracking can remove occlusions and enable 3D avatar body reconstruction by avoiding obstacles and generating high-resolution, rich-information multi-view (RGB) photos. Training failures of 3D avatar reconstruction techniques lead to distortions and loss of features in 3D reconstructed models due to several reasons, including limited viewpoint coverage, visible occlusions, and texture disappearance. The recently developed end-to-end trainable deep neural network technique This work presents PIXGAN-Drone, a photo-realistic 3D avatar reconstruction system for the human body from multi-view photos. To create high-resolution 2D models, is predicated on integrating aerial photography sensors (a steady autonomous circular motion system) coupled with active tracking drones into the Pix2Pix GANs training framework. Accurate and realistic 3D models can be created with conditional image-to-image translation and dynamic aerial views. This study used tests on several datasets to show that our approach outperforms state-of-the-art approaches for a variety of metrics (Chamfer, P2S, and CED). Our 3D reconstructed human avatars in RenderPeople were 0.0293, 0.0271, and 0.0232; on People Snapshot (inside), 0.0133, 0.0136, 0.0050; on People Snapshot (outdoor), 0.0154, 0.0101, 0.0063; and on Custom data-drone (collected dataset), 0.0316, 0.0275, 0.0216.

Keywords

• 3D human avatar
• 3D reconstruction
• PIX2PIXGAN
• body model rendering
• drone active tracking