Dual Prototype Learning for Few Shot Semantic Segmentation

Abstract

Few-shot segmentation (FSS) is a challenging task because the same class of targets in the support and query images may have different scales, textures and background information. Prototype learning (PL) is a current mainstream FSS method, which characterizes the interaction between the prototype vector and query feature. However, the prototype vector commonly based on global average pooling only contains first-order feature information, which is vulnerable to varying appearance of similar target and the diversity of background. Moreover, the auxiliary information of the query image is not fully explored in previous prototype learning methods. In this paper, we propose a dual prototype learning (DPL) based on a second-order prototype (SOP) and self-support first-order prototype with a constraint mechanism (SSFPC) to improve the FSS performance. The SOP can capture higher-order statistical information by averaging the covariance matrix of the feature map. The similarity between the first-order support prototype and the first-order self-support query prototype is introduced to boost the adaptability of the first-order prototype to the query image. The remarkable performance gains on the benchmarks (PASCAL- ${5^{i}}$ and COCO- ${20^{i}}$ ) manifest the effectiveness of our method. Our source code will be available at https://github.com/13ww/DPL.git.

Keywords

• Few-shot learning
• semantic segmentation
• first-order prototype
• second-order prototype