Multimodal Chinese Agricultural News Classification Method Based on Interactive Attention

Abstract

Most current research on Chinese agricultural news is limited to text analysis and seldom integrates images, leading to a scarcity of multimodal Chinese agricultural news datasets and an evident gap in multimodal Chinese agricultural news research. To address this, we propose the VECO method, a novel multimodal Chinese agricultural news classification approach that leverages interactive attention mechanisms. This algorithm uses ERNIE for text feature extraction and ViT(Vision Transformer) for image feature extraction, focusing on the interplay of features across modalities to uncover the congruent emotional content present in both the images and text. The integrated features are merged with individual image and text features and subsequently processed through a softmax layer to determine the classification outcomes. Our experiments, conducted on an in-house multimodal Chinese agricultural news dataset, demonstrate that the VECO method outperforms the baseline model, with improvements of 3.27% in precision, 0.59% in recall, and 1.92% in f1-score. The multimodal classification of Chinese agricultural news yields superior performance compared to text-only classification, and the results of the VECO model are notably better than those of other multimodal classification models. Future research can focus on optimizing the multimodal feature fusion algorithm to adapt to more complex agricultural news scenarios.

Keywords

• Multimodal learning
• multimodal classification
• multimodal Chinese agricultural news dataset
• interactive attention mechanism
• attention mechanism
• feature fusion