DualPathGAN: Facial reenacted emotion synthesis

Abstract

Abstract Facial reenactment has developed rapidly in recent years, but few methods have been built upon reenacted face in videos. Facial‐reenacted emotion synthesis can make the process of facial reenactment more practical. A facial‐reenacted emotion synthesis method is proposed that includes a dual‐path generative adversarial network (GAN) for emotion synthesis and a residual‐mask network to impose structural restrictions to preserve the mouth shape of the source person. To train the dual‐path GAN more effectively, a learning strategy based on separated discriminators is proposed. The method is trained and tested on a very challenging imbalanced dataset to evaluate the ability to deal with complex practical scenarios. Compared with general emotion synthesis methods, the proposed method can generate more realistic facial emotion synthesised images or videos with higher quality while retaining the expression contents of the original videos. The DualPathGAN achieves a Fréchet inception distance (FID) score of 9.20, which is lower than the FID score of 11.37 achieved with state‐of‐the‐art methods.