GroupPrompter: A Prompting Method for Semantic Segmentation Based on SAM

Abstract

The SAM shows remarkable generalization and transformable capabilities for category-agnostic segmentation. Although the semantics in latent space are explored slightly, more researches are working on instance segmentation. And it’s unclear how to design the appropriate prompts for semantic segmentation, which have large impact the performance. In this paper, by summarizing several existing methods for semantic segmentation with SAM, we propose a learnable prompt method for semantic segmentation based on the SAM model, incorporating the latent semantic features with prompt learning by a grouping approach, referred as GroupPrompter. This enables SAM to perform semantic segmentation with the automatic learned prompts. And the experimental results on ADE20K, Pascal Context and COCO-Stuff datasets validate the effectiveness of the proposed method.

Keywords

• Foundation model
• prompt learning
• SAM
• semantic segmentation
• visual grouping