The deep spatiotemporal network with dual-flow fusion for video-oriented facial expression recognition

Abstract

The video-oriented facial expression recognition has always been an important issue in emotion perception. At present, the key challenge in most existing methods is how to effectively extract robust features to characterize facial appearance and geometry changes caused by facial motions. On this basis, the video in this paper is divided into multiple segments, each of which is simultaneously described by optical flow and facial landmark trajectory. To deeply delve the emotional information of these two representations, we propose a Deep Spatiotemporal Network with Dual-flow Fusion (defined as DSN-DF), which highlights the region and strength of expressions by spatiotemporal appearance features and the speed of change by spatiotemporal geometry features. Finally, experiments are implemented on CK+ and MMI datasets to demonstrate the superiority of the proposed method. Keywords: Facial expression recognition, Deep spatiotemporal network, Optical flow, Facial landmark trajectory, Dual-flow fusion.