Transformer-Based Dual-Branch Multiscale Fusion Network for Pan-Sharpening Remote Sensing Images

Abstract

Due to the limitations of satellite sensors, we can only obtain MS images and PAN images separately. The focus of our attention is to utilize the pan-sharpening method to generate the high-resolution multispectral (HRMS) images. In this article, we proposed the dual-branch multiscale fusion network, which based on the spatial-spectral transformer to comprehensively capture the information contained in MS images and PAN images at different scales. The architecture of our network consists of three parts: during the feature extraction and image fusion stage, we first independently apply upscaling and downscaling operations to the MS and PAN images. Subsequently, we concatenate the images from the two distinct branches and input them into the shallow feature extraction module individually. And then we input them into our adaptive feature extraction block to further extract the crucial details of the images using the attention mechanism. The images a various scales in different branches are then passed through three spectral transformer and three spatial transformer modules to perform a comprehensive extraction of both spatial and spectral characteristics. Finally, the residual local feature module is utilized during the image reconstruction part to deeply extract intricate information from the images and obtain the final HRMS fused image. We have conducted both simulated and real experiments on the benchmark datasets QB and WV2. The final qualitative and quantitative comparative results demonstrate that our innovative method outperforms the current SOTA methods.

Keywords

• Attention mechanism
• convolutional neural network
• pansharpening
• transformer