GFA: Generalized Feature Affinity for Semantic Segmentation in Resolution-Degraded Images

Abstract

In the remote sensing field, resolution-degraded images are often captured under poor imaging conditions, leading to missing texture information for segmentation. To tackle these challenges, a parallel super-resolution branch can be incorporated into the segmentation pipeline to restore detailed features through a feature supplementation operation. However, existing methods based on the structure are mainly limited by the direct supplementation operation between the two tasks with different feature distributions, and the original semantic distribution may be damaged. Therefore, a novel learning method called generalized feature affinity (GFA) is proposed. It is expected to realize more comprehensive feature supplementation in a consistent feature distribution space between the two tasks. Specifically, a global distribution affinity module is developed to encourage uniformity in the intraclass distribution between restored and semantic features. Then, a local texture affinity module is designed to better transfer detailed information by enhancing the texture prominence in feature maps and exploring supplementation in spatial and channel aspects. To guide the supplementation direction more effectively, a results-oriented hard sampling strategy, which integrates auxiliary spatial attention is further proposed to enhance the performance of the aforementioned modules. Extensive experiments are conducted on the widely recognized benchmarks ISPRS Potsdam and Vaihingen, and LoveDA datasets. The proposed method results in an improvement of about 2% mIoU across many commonly used segmentation models, demonstrating the effectiveness and generalization of GFA.

Keywords

• Generalized feature affinity (GFA)
• semantic segmentation (SS)
• super-resolution (SR)