E2‐capsule neural networks for facial expression recognition using AU‐aware attention

Abstract

Capsule neural network is a new and popular technique in deep learning. However, the traditional capsule neural network does not extract features sufficiently before the dynamic routing between capsules. In this study, one double enhanced capsule neural network (E2‐Capsnet) that uses AU‐aware attention for facial expression recognition (FER) is proposed. The E2‐Capsnet takes advantage of dynamic routing between capsules and has two enhancement modules which are beneficial to FER. The first enhancement module is the convolutional neural network with AU‐aware attention, which can focus on the active areas of the expression. The second enhancement module is the capsule neural network with multiple convolutional layers, which enhances the ability of the feature representation. Finally, the squashing function is used to classify the facial expression. The authors demonstrate the effectiveness of E2‐Capsnet on the two public benchmark datasets, RAF‐DB and EmotioNet. The experimental results show that their E2‐Capsnet is superior to the state‐of‐the‐art methods. The code is available at https://github.com/ShanCao18/E2‐Capsnet.

Keywords

• AU‐aware attention
• traditional capsule neural network
• dynamic routing
• double enhanced capsule neural network
• facial expression recognition
• enhancement module