Self‐supervised non‐rigid structure from motion with improved training of Wasserstein GANs

Abstract

Abstract This study proposes a self‐supervised method to reconstruct 3D limbic structures from 2D landmarks extracted from a single view. The loss of self‐consistency can be reduced by performing a random orthogonal projection of the reconstructed 3D structure. Thus, the training process can be self‐supervised by using geometric self‐consistency in the reconstruction–projection–reconstruction process. The self‐supervised network mainly consists of graph convolution and Transformer encoders. This network is called the SS‐Graphformer. By adding a discriminator, the SS‐Graphformer is used as a generator to form a Wasserstein Generative Adversarial Network architecture with a Gradient Penalty to improve the accuracy of the reconstruction. It is experimentally demonstrated that the addition of the 2D structure discriminator can significantly improve the accuracy of the reconstruction.

Keywords

• computer vision
• neural nets