Multi-Level Cross-Modal Interactive-Network-Based Semi-Supervised Multi-Modal Ship Classification

Abstract

Ship image classification identifies the type of ships in an input image, which plays a significant role in the marine field. To enhance the ship classification performance, various research focuses on studying multi-modal ship classification, which aims at combining the advantages of visible images and infrared images to capture complementary information. However, the current methods simply concatenate features of different modalities to learn complementary information, which neglects the multi-level correlation between different modalities. Moreover, the existing methods require a large amount of labeled ship images to train the model. How to capture the multi-level cross-modal correlation between unlabeled and labeled data is still a challenge. In this paper, a novel semi-supervised multi-modal ship classification approach is proposed to solve these issues, which consists of two components, i.e., multi-level cross-modal interactive network and semi-supervised contrastive learning strategy. To learn comprehensive complementary information for classification, the multi-level cross-modal interactive network is designed to build local-level and global-level cross-modal feature correlation. Then, the semi-supervised contrastive learning strategy is employed to drive the optimization of the network with the intra-class consistency constraint based on supervision signals of unlabeled samples and prior label information. Extensive experiments on the public datasets demonstrate that our approach achieves state-of-the-art semi-supervised classification effectiveness.

Keywords

• ship classification
• deep multi-modal learning
• semi-supervised learning