Gated Fusion Network With Progressive Modality Enhancement for Sentiment Analysis

Abstract

Multi-modal sentiment analysis aims to recognize the sentiment expressed by the studied subjects utilizing textual, visual, and acoustic cues. Given the vast application prospects of multi-modal sentiment analysis in domains such as human-computer interaction and intelligent customer service systems, it has gained widespread attention in recent years. The design of fusion strategy significantly impacts analysis results. Different modalities varied on the extent of contributions to the sentiment analysis process, and the textual modality tends to play a crucial role. However, most existing fusion strategies have ignored the issue of uneven distribution of sentiment information across modalities. Therefore, in this study, we introduce a gated fusion network with progressive modality enhancement. To effectively address the challenge of imbalanced sentiment distribution, we propose a progressive modality enhancement approach that facilitates the interaction of sentiment cues. Furthermore, to overcome the limitations of insufficient inter-modal interactions observed in previous works, we introduce a gated multi-modal information interaction module. This module comprehensively exploits multi-modal data, ensuring that each modality can effectively utilize information from others. Experimental results demonstrate that the proposed method achieves highly competitive performance on the CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets. Furthermore, the ablation experiments confirm the dominant role of the textual modality, while indicating that the visual modality generally performs better than the acoustic modality across our experimental settings.

Keywords

• Sentiment analysis
• multi-modal fusion
• modality enhancement