Efficient Pansharpening by Joint-Modality Recursive Training

Abstract

Multispectral images captured by remote sensing systems usually have low spatial resolution. Pansharpening offers a promising solution by enhancing the resolution of these low-resolution multispectral images to a high-resolution multispectral without the need for costly hardware upgrades. Existing methods employ either CNN or Transformer as the feature extractor backbone, however, CNN-based methods are weak in capturing long-distance correlation, and Transformer-based methods are limited to extracting fine-grain detail. Moreover, these models achieve impressive results with numerous learnable parameters, which makes them impractical for integration into remote sensing systems. In this work, a parameter-efficient pansharpening model, named joint-modality association network, is built by leveraging complementary information from multiple modalities and recursive training. It aims to improve the resolution of remote-sensing images. Specifically, we efficiently leverage the complementary information from different modalities, including the transformer and CNN joint block, and employ a hierarchical association mechanism to create a more distinctive and informative representation by associating intramodality and cross-modality. Furthermore, the parameter-sharing mechanism of recursive training can effectively reduce the number of parameters in the model. Benefiting from its lightweight design and effective information fusion strategy, the proposed method can generate faithful super-resolved multispectral images that excel in both spectral and spatial resolution. Experimental results show the superiority of the proposed method over extensive benchmarks.

Keywords

• Multimodalities
• multispectral images
• pansharpening
• parameter-efficient
• spatial resolution