Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation

by   Tianle Chen, et al.

Weakly Supervised Semantic Segmentation (WSSS) with only image-level supervision has garnered increasing attention due to its low annotation cost compared to pixel-level annotation. Most existing methods rely on Class Activation Maps (CAM) to generate pixel-level pseudo labels for supervised training. However, it is well known that CAM often suffers from partial activation – activating the most discriminative part instead of the entire object area, and false activation – unnecessarily activating the background around the object. In this study, we introduce a simple yet effective approach to address these limitations by harnessing the recently released Segment Anything Model (SAM) to generate higher-quality pseudo labels with CAM. SAM is a segmentation foundation model that demonstrates strong zero-shot ability in partitioning images into segments but lacks semantic labels for these regions. To circumvent this, we employ pseudo labels for a specific class as the signal to select the most relevant masks and label them to generate the refined pseudo labels for this class. The segments generated by SAM are highly precise, leading to substantial improvements in partial and false activation. Moreover, existing post-processing modules for producing pseudo labels, such as AffinityNet, are often computationally heavy, with a significantly long training time. Surprisingly, we discovered that using the initial CAM with SAM can achieve on-par performance as the post-processed pseudo label generated from these modules with much less computational cost. Our approach is highly versatile and capable of seamless integration into existing WSSS models without modification to base networks or pipelines. Despite its simplicity, our approach improves the mean Intersection over Union (mIoU) of pseudo labels from five state-of-the-art WSSS methods by 6.2% on average on the PASCAL VOC 2012 dataset.


page 2

page 7

page 8

page 9


Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach

Weakly supervised semantic segmentation is a challenging task as it only...

Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation with weak labels is a long-lived...

Exploring Unsupervised Cell Recognition with Prior Self-activation Maps

The success of supervised deep learning models on cell recognition tasks...

Threshold Matters in WSSS: Manipulating the Activation for the Robust and Accurate Segmentation Model Against Thresholds

Weakly-supervised semantic segmentation (WSSS) has recently gained much ...

Out-of-Candidate Rectification for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation is typically inspired by class a...

An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems

The Segment Anything Model (SAM) has demonstrated exceptional performanc...

Multi-Granularity Denoising and Bidirectional Alignment for Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation (WSSS) models relying on class a...

Please sign up or login with your details

Forgot password? Click here to reset