Multistream STGAN: A Spatiotemporal Image Fusion Model With Improved Temporal Transferability

Abstract

Spatiotemporal satellite image fusion aims to generate remote sensing images satisfying both high spatial and temporal resolution by integrating different satellite imagery datasets with distinct spatial and temporal resolutions. Such fusion technique is crucial for numerous applications that require frequent monitoring at fine spatial and temporal scales spanning agriculture, environment, natural resources, and disaster management. However, existing fusion models have difficulty accommodating abrupt spatial changes in land cover among images and dealing with temporally distant image data. This article proposes a novel multistream spatiotemporal fusion generative adversarial network (STGAN) model for spatiotemporal satellite image fusion that can produce accurate fused images and accommodate substantial temporal differences between the input images. The STGAN employs a conditional generative adversarial network architecture with a multistream input design to better learn temporal features. The generator of STGAN comprises convolutional blocks, a spatial transformer module, a channel attention network, and a U-net module designed to better capture spatial and temporal features from the multiresolution input images. Comprehensive evaluations of the proposed STGAN model have been performed on the Coleambally Irrigation Area and Lower Gwydir Catchment datasets, using both visual inspection and spatial and spectral metrics, including root mean square error, relative dimensionless global error synthesis, spectral angle mapping, structural similarity index measure, and local binary pattern. The experiments show that the proposed STGAN model consistently outperforms existing benchmark models and is capable of generating high-quality fused remote sensing data product of high spatial and temporal resolution.

Keywords

• Deep learning
• generative adversarial network (GAN)
• remote sensing
• spatiotemporal image fusion