Adaptive spatio-temporal attention neural network for crossdatabase micro-expression recognition

Abstract

Background: Recognizing human emotions by micro-expression recognition is one of the most critical issues in human-computer interaction applications. Cross-database micro-expression recognition (CDMER) is an increasingly significant problem in micro-expression recognition and analysis in recent years. Since the training data and testing data in CDMER come from different micro-expression databases, CDMER is more challenging than the conventional micro-expression recognition. Methods: In this paper, an Adaptive Spatio-Temporal Attention Neural Network (ASTANN) using attention mechanism is presented to deal with this challenging and critical problem. To this end, the micro-expression database SMIC and CASME II are firstly preprocessed by optical flow approach, which extract motion information among video frames that represents discriminative features of micro-expression. After preprocessing, a novel adaptive framework with spatio-temporal attention module is designed to assign spatial and temporal weights to enhance the most discriminative features. Then the deep neural network extracts cross-domain feature, in which the second-order statistics of the sample features in source domain is aligned with the ones in the target domain by minimizing the correlation alignment (CORAL) loss such that source and target database share similar distributions. Results: To evaluate the performance of ASTANN, experiments are conducted based on SMIC and CASME II databases under a standard experimental evaluation protocol of CDMER. The experimental results demonstrate that ASTANN outperforms other methods in relevant cross-database tasks. Conclusions: Extensive experiments are conducted on the benchmark tasks and results show that ASTANN achieves the best performance over all other approaches. This remarkable performance demonstrates the superiority of our method for solving CDMER problem.

Keywords

• Cross-database micro-expression recognition
• Deep learning
• Attention mechanism
• Domain adaption