Multispectral Pansharpening Based on Multisequence Convolutional Recurrent Neural Network

Abstract

Multispectral (MS) pansharpening is defined as the fusion of spatial information in panchromatic (PAN) image and spectral information in MS image. In this work, we propose an MS pansharpening based on multisequence convolutional recurrent neural network (MCRNN). The proposed MCRNN contains two subnetworks (shallow feature extraction subnetwork and deep feature fusion subnetwork). In the shallow feature extraction subnetwork, PAN and MS images are superimposed in the spectral dimension as multisequence data. A convolutional neural network based on residual learning is then used to obtain the feature maps from multisequence data. In the deep feature fusion subnetwork, since MS and PAN images are highly correlated, a convolutional recurrent neural network belonging to recurrent neural network is used to model adjacent and across-band relationships between these feature maps to capture the local and global correlations of the features in different bands. The global average pooling is then performed on the output results to yield the pansharpening result. Several datasets are tested at reduced and full-resolution experiments, the experimental results show that the performance of the proposed MCRNN is superior to the traditional pansharpening methods.

Keywords

• Convolutional neural network (CNN)
• feature extraction
• feature fusion
• multispectral (MS) pansharpening
• recurrent neural network (RNN)