Deep Representation Learning for Multimodal Emotion Recognition Using Physiological Signals

Abstract

Physiological signal analysis has gained a lot of interest in recent years and has been used in a variety of fields including emotion recognition, activity recognition, and health monitoring. However, emotion recognition based on physiological signals is not yet explored entirely using deep learning, and there are still some exciting challenges to be handled. For example, deep representation learning for spatio-temporal feature extraction, the discrimination between adjacent emotions with entangled features, and the imbalanced distribution of data are the most prominent issues in emotion recognition. This work focuses on deep multimodal representation learning of physiological signals to alleviate the aforementioned challenges. We introduce a novel deep learning architecture for emotion classification that effectively extracts spatio-temporal information from physiological signals. We proposed a mutual attention mechanism to extract emotion-specific features for improved classification. To handle the issue of adjacent emotions and imbalanced data, we introduce a dense max-margin loss function based on Gaussian similarity measure. Our experiments on different datasets reveal that the proposed emotion classification methodology effectively learns a balanced deep representation of physiological signals, significantly maximizes the inter-class margin, and reduces intra-class variance to discriminate between different classes of emotions.

Keywords

• Emotion recognition
• deep learning
• attention mechanism
• imbalanced data
• EEG
• ECG