A Survey on the Deep Learning-Based Mismatch Removal: Principles and Methods

Abstract

Due to the inherent limitations of matching algorithms and the complexities associated with image contents, mismatches are inevitable and can have detrimental effects on downstream tasks in computer vision and remote sensing. Researchers have published numerous reviews on mismatch removal, which may suffer from two primary deficiencies. Firstly, these reviews are often embedded within studies that primarily focus on image matching, thereby limiting the detailed and comprehensive analysis of mismatch removal methods. Secondly, reviews of deep learning (DL)-based methods, despite their numerous existence and interconnection, tend to be fragmentary and lack a systematic approach. To address these two shortcomings, this paper presents a comprehensive survey of DL-based mismatch removal principles and methods. We provide a summary of network architectures, techniques for extracting geometrical information, and various training modes. Specifically, we highlight the importance of permutation invariance in mining operations, enumerate a majority of existing mining methods, and provide an explanation of their permutation invariant properties. Furthermore, we present both the intuitive motivation and mathematical analysis of commonly used methods, elucidating their underlying principles and efficacy. In the conclusion, we predict upcoming trends based on the findings of our review, aiming to provide valuable insights into mismatch removal techniques and guide their practical applications.

Keywords

• Image matching
• mismatch removal
• deep learning
• geometrical information mining
• permutation invariant