Learning to Find the Optimal Correspondence Between SAR and Optical Image Patches

Abstract

This study addresses the problem of finding the optimal correspondence for a given synthetic aperture radar (SAR) image patch from a large collection of optical reference patches, which is crucial for various applications, including remote sensing, place recognition, and aircraft navigation. However, achieving one-to-one SAR-Optical patch correspondence is challenging due to the distinct modal discrepancy and the poor discriminability of the target instances. To address these challenges, we propose a cross-modal patch correspondence scheme that consists of two modules: A retrieval-based coarse search module and a correspondence refinement module. Specifically, to explicitly represent the modal discrepancy, we first introduce a cross-modal adversarial learning strategy in the coarse search module and learn the modal-invariant feature embedding for retrieval. Furthermore, to improve the instance discriminability of retrieved candidates, we propose a graph representation in the refinement module to integrate the visual and spatial information, which is finally fed to an attention graph network to estimate the optimal correspondence. To evaluate the effectiveness of the proposed scheme, we also propose three new SAR-Optical patch correspondence datasets. Comprehensive experiments show that our approach significantly outperforms the competitors on all three datasets.

Keywords

• Adversarial training
• cross-modal image retrieval
• graph neural network
• synthetic aperture radar (SAR)