Multi-modal physiological signal emotion recognition based on 3D hierarchical convolution fusion

Abstract

In recent years, physiological signals such as electroencephalograhpy (EEG) have gradually become popular objects of emotion recognition research because they can objectively reflect true emotions.However, the single-modal EEG signal has the problem of incomplete emotional information representation, and the multi-modal physiological signal has the problem of insufficient emotional information interaction.Therefore, a 3D hierarchical convolutional fusion model was proposed, which aimed to fully explore multi-modal interaction relationships and more accurately describe emotional information.The method first extracted the primary emotional representation information of EEG , electro-oculogram (EOG) and electromyography (EMG) by depthwise separable convolution network, and then performed 3D convolution fusion operation on the obtained multi-modal primary emotional representation information to realize the pairwise mode local interactions between states and global interactions among all modalities, so as to obtain multi-modal fusion representations containing emotional characteristics of different physiological signals.The results show that the accuracy in the valence and arousal of the two-class and four-class tasks on DEAP dataset are both 98% by the proposed model.

Keywords

• physiological signal;emotion recognition;3D hierarchical convolutional;multi-modal interaction