Landmark-Based Adaptive Graph Convolutional Network for Facial Expression Recognition

Abstract

Facial expression recognition (FER) is an important task in human emotion analysis. Though researchers have made great progress in FER, the semantic information lying in facial landmarks has not been fully exploited. In this paper, we propose a landmark-based adaptive graph convolutional network (LBAGCN) for FER. It takes facial landmarks as input in both phases of model training and inference and thus preserves sensitive privacy information well. Based on the facial landmark sequence extracted from a video clip, we define landmark motion sequence, muscle sequence and muscle motion sequence. Correspondingly, we construct four facial graphs with learnable topologies, named landmark graph, landmark motion graph, muscle graph and muscle motion graph. Each of the facial graphs contains useful semantic information relevant to facial expressions. The proposed LBAGCN consists of four parallel adaptive graph convolutional networks (AGCNs), each of which employs well-designed adaptive graph convolution blocks and multi-branch temporal convolutional network blocks to learn the spatial and temporal features from a facial graph automatically. The class scores of AGCNs are fused to boost the FER accuracy. Extensive experiments have been carried out to validate the proposed LBAGCN on the recognized CK+ dataset and Oulu CASIA dataset. Compared with state-of-the-art methods, the proposed LBAGCN achieves competitive FER accuracy especially on the challenging Oulu CASIA dataset. Testing results on practical platforms confirm that the proposed LBAGCN can be a feasible solution for real-time FER applications.

Keywords

• Adaptive graph convolutional network
• facial expression recognition
• facial graph representation
• multi-stream network