Rethinking Semantic Segmentation: A Prototype View

03/28/2022
by   Tianfei Zhou, et al.
0

Prevalent semantic segmentation solutions, despite their different network designs (FCN based or attention based) and mask decoding strategies (parametric softmax based or pixel-query based), can be placed in one category, by considering the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, this study uncovers several limitations of such parametric segmentation regime, and proposes a nonparametric alternative based on non-learnable prototypes. Instead of prior methods learning a single weight/query vector for each class in a fully parametric manner, our model represents each class as a set of non-learnable prototypes, relying solely on the mean features of several training pixels within that class. The dense prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space, by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to handle arbitrary number of classes with a constant amount of learnable parameters. We empirically show that, with FCN based and attention based segmentation models (i.e., HRNet, Swin, SegFormer) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework yields compelling results over several datasets (i.e., ADE20K, Cityscapes, COCO-Stuff), and performs well in the large-vocabulary situation. We expect this work will provoke a rethink of the current de facto semantic segmentation model design.

READ FULL TEXT

page 3

page 5

page 7

page 15

page 16

page 17

08/18/2019

PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment

Despite the great progress made by deep CNNs in image semantic segmentat...
09/15/2022

Visual Recognition with Deep Nearest Centroids

We devise deep nearest centroids (DNC), a conceptually elegant yet surpr...
10/18/2022

Number-Adaptive Prototype Learning for 3D Point Cloud Semantic Segmentation

3D point cloud semantic segmentation is one of the fundamental tasks for...
01/15/2022

Prototype Guided Network for Anomaly Segmentation

Semantic segmentation methods can not directly identify abnormal objects...
03/23/2022

StructToken : Rethinking Semantic Segmentation with Structural Prior

In this paper, we present structure token (StructToken), a new paradigm ...
06/25/2020

Fully Convolutional Open Set Segmentation

In semantic segmentation knowing about all existing classes is essential...
10/05/2022

GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

Prevalent semantic segmentation solutions are, in essence, a dense discr...

Please sign up or login with your details

Forgot password? Click here to reset