Denoising Diffusion Semantic Segmentation with Mask Prior Modeling

by   Zeqiang Lai, et al.

The evolution of semantic segmentation has long been dominated by learning more discriminative image representations for classifying each pixel. Despite the prominent advancements, the priors of segmentation masks themselves, e.g., geometric and semantic constraints, are still under-explored. In this paper, we propose to ameliorate the semantic segmentation quality of existing discriminative approaches with a mask prior modeled by a recently-developed denoising diffusion generative model. Beginning with a unified architecture that adapts diffusion models for mask prior modeling, we focus this work on a specific instantiation with discrete diffusion and identify a variety of key design choices for its successful application. Our exploratory analysis revealed several important findings, including: (1) a simple integration of diffusion models into semantic segmentation is not sufficient, and a poorly-designed diffusion process might lead to degradation in segmentation performance; (2) during the training, the object to which noise is added is more important than the type of noise; (3) during the inference, the strict diffusion denoising scheme may not be essential and can be relaxed to a simpler scheme that even works better. We evaluate the proposed prior modeling with several off-the-shelf segmentors, and our experimental results on ADE20K and Cityscapes demonstrate that our approach could achieve competitively quantitative performance and more appealing visual quality.


page 1

page 3

page 4

page 5

page 8

page 16

page 17


DFormer: Diffusion-guided Transformer for Universal Image Segmentation

This paper introduces an approach, named DFormer, for universal image se...

Generative Semantic Segmentation

We present Generative Semantic Segmentation (GSS), a generative learning...

Label-Efficient Semantic Segmentation with Diffusion Models

Denoising diffusion probabilistic models have recently received much res...

CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion Models

Camouflaged Object Detection (COD) is a challenging task in computer vis...

Multi-Class Segmentation from Aerial Views using Recursive Noise Diffusion

Semantic segmentation from aerial views is a vital task for autonomous d...

Diffusion models as plug-and-play priors

We consider the problem of inferring high-dimensional data 𝐱 in a model ...

Masked Diffusion as Self-supervised Representation Learner

Denoising diffusion probabilistic models have recently demonstrated stat...

Please sign up or login with your details

Forgot password? Click here to reset