Multimodal Fake News Detection Based on Contrastive Learning and Similarity Fusion

Abstract

In social media, the errors and harm caused by multimodal fake news are becoming increasingly serious. In the detection of multimodal fake news, special attention is needed to the problem of insufficient utilization of fine-grained intra- and inter-modal information and the ineffective integration of this information during the fusion process. Our study primarily aims to address this problem by proposing a multimodal fake news detection model based on contrastive learning and similarity fusion. First, Image and text features, enhanced using a multimodal bidirectional attention mechanism, are aligned in the semantic space. Then, contrastive learning, which fuses the unimodal prediction results, is utilized for a fine-grained understanding of the features. Finally, similarity scores guide the generation of gating weights to dynamically adjust the contribution of each modality, resulting in the final multimodal features for fake news detection. Experiments conducted on two real-world datasets show that the model improves accuracy by 3.6% and 2.4% compared to the baseline model, effectively enhancing the model’s ability to detect multimodal fake news.

Keywords

• Fake news
• multimodal
• contrastive learning
• feature fusion