Pan-Sharpening Based on Convolutional Neural Network by Using the Loss Function With No-Reference

Abstract

In order to preserve the spatial and spectral information of the original panchromatic and multispectral images, this article designs a loss function suitable for pan-sharpening and a four-layer convolutional neural network that could adequately extract spectral and spatial features from original source images. The major advantage of this study is that the designed loss function does not need the reference fused image, and then the proposed pan-sharpening method does not need to make the simulation data for training. This is the big difference from most existing pan-sharpening methods. Moreover, the loss function takes into account the characteristics of remote sensing images, including the spatial and spectral evaluation indicators. We also add the feature enhancement layer in convolutional neural network, thus, the proposed four-layer network contains feature extraction, feature enhancement, linear mapping and reconstruction. In order to evaluate the effectiveness and universality of the proposed fusion model, we selected thousands of remote sensing images that include different sensors, different times and different land-cover types to make the training dataset. By evaluating the performance on the WorldView-2, Pleiades and Gaofen-1 experimental data, the results show that the proposed method achieves optimal performance in terms of both the subjective visual effect and the object assessment. Furthermore, the codes will be available at https://github.com/Zhangxi-Xiong/pan-sharpening.

Keywords

• Convolutional neural network
• deep learning
• feature enhancement
• loss function
• pan-sharpening
• remote sensing image fusion