NVS-GAN: Benefit of generative adversarial network on novel view synthesis

Abstract

The methodology to generate new views for an object from provided input object view is called Novel View Synthesis (NVS). Humans imagine novel views through prior knowledge gathered through their lifetime. NVS-GAN predicts the novel views through computation. Literature survey reveals that there are limited NVS models with low Trainable Parameter Count (TPC) and low model size. Also, a study on the effect of different loss functions on NVS models was lacking. Lowering the TPC indicates less computational steps for the model to predict the output, therefore desirable. Combined with a low model size, the proposed model will become more suitable for deployment in diverse devices having limited resources for computation. Application of right combination of loss functions yield better accuracy. To address these research gaps, NVS-GAN is proposed. NVS-GAN is a Generative Adversarial Network (GAN) approach which yields NVS-Generator which performs NVS. NVS-Generator incorporates identity skip connections, bilinear sampling module, Depthwise Separable Convolution (DSC) as design features and results in low TPC, model size. In addition to discriminator loss, NVS-GAN is trained with different combinations of loss functions i.e. Mean Absolute Error (MAE) loss, Structural Similarity Index Measure (SSIM) loss, Huber loss on chair and car objects of ShapeNet dataset. The performance of NVS-Generator on test set measured in terms of MAE and SSIM is tabulated and analysed. The performance is compared with existing NVS models. The proposed NVS-GAN experiment recorded reduction in NVS-Generator TPC in 37 %–54.6 % range and reduction in model size between 37.2 % and 47.6 % range. NVS-Generator reduced MAE upto 55 % and improved SSIM upto 4 % than existing models. Summarily, NVS-GAN increased model performance and made the model “lightweight”.

Keywords

• NVS-GAN
• NVS-Generator
• Depthwise separable convolution
• Skip connections