Spectral–Spatial Score Fusion Attention Network for Hyperspectral Image Classification With Limited Samples

Abstract

Convolutional neural network (CNN) and transformer-based models have been widely used in hyperspectral image (HSI) classification due to their excellent local and global modeling capabilities. In addition, attention mechanism is widely embedded in these models due to the effective enhancement of features learning. However, it is difficult to learn adaptive weights that effectively enhance features and most of existing methods lack transitional processing of shallow features. To overcome the above issues, a lightweight spectral–spatial score fusion attention network (S3FAN) with dual architecture is proposed for HSI classification with limited samples. Different from the regular dual branch models, S3FAN first performs pixel-level interaction and spatial feature extraction, then the obtained two sets of features are weighted and fused. In addition, we designed a spectral–spatial score fusion attention mechanism to enhance dynamic attention to spectral–spatial features. We also propose a spectral transition block to enhance model adaptability. Performance evaluation experiments conducted on five HSI datasets demonstrate that S3FAN has higher accuracy and generalization capabilities compared to existing advanced CNN and Transformer-based methods, with improvements in terms of OA around 3.18%–34.3% for Indian Pines, 5.87%–28.58% for University of Pavia, 2.57%–15.37% for Salinas, 1.64%–8.95% for Yellow River Delta, 2.87%–11.33% for WHU-Hi-LongKou, under ten samples per class.

Keywords

• Hyperspectral image (HSI) classification
• limited samples
• spectral–spatial attention (SSA)
• transformer