Context-Driven Automatic Target Detection With Cross-Modality Real-Synthetic Image Merging

Abstract

This article presents pioneer research on joint scene-target analysis and proposes novel cross-modality real-synthetic target feature fusion method. To begin, multisensor remote sensing images are jointly leveraged for geographical region classification. After that, a novel Context-Aware Region Masking and Situation AWareness (CARMSAW) strategy is employed for target classification based on the inherent target properties and capabilities reflected by SAR and infrared (IR) imagery, and the cross-modality Real-synthetic Image Merging (CRIM) strategy is employed for feature enhancement. Specifically, to tackle with the random deviations of the real SAR imagery from the ideal ones, the synthetic SAR signature generated based on the target CAD model is treated as a “skeleton” with known structure for real-sync target feature alignment. To facilitate the recognition of aircrafts, we leverage on the IR images to construct an “exoskeleton” for the target SAR signature, so that the dimension/shape/contour of the target and its electromagnetic features are united. Furthermore, we propose a novel color-guided component-level attention mechanism, in which the SAR image is partitioned into several subregions highlighted or blacked-out adaptively based on their significance level. To demonstrate the effectiveness of the proposed CARMSAW strategy, a series of experiments are carried out based on the SAR-optical image pairs from the SEN1-2 dataset, the SpaceNet6 dataset, and a self-constructed ship detection dataset featuring the Port of Rotterdam. To verify the performance the proposed CRIM method, experiment results based on both the self-constructed SAR-IR dataset and the MSTAR-SAMPLE dataset in the public domain are provided.

Keywords

• Multimodal data fusion
• target detection
• unmanned aerial vehicles