Rethinking Semantic Segmentation: A Prototype View

by   Tianfei Zhou, et al.

Prevalent semantic segmentation solutions, despite their different network designs (FCN based or attention based) and mask decoding strategies (parametric softmax based or pixel-query based), can be placed in one category, by considering the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, this study uncovers several limitations of such parametric segmentation regime, and proposes a nonparametric alternative based on non-learnable prototypes. Instead of prior methods learning a single weight/query vector for each class in a fully parametric manner, our model represents each class as a set of non-learnable prototypes, relying solely on the mean features of several training pixels within that class. The dense prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space, by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to handle arbitrary number of classes with a constant amount of learnable parameters. We empirically show that, with FCN based and attention based segmentation models (i.e., HRNet, Swin, SegFormer) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework yields compelling results over several datasets (i.e., ADE20K, Cityscapes, COCO-Stuff), and performs well in the large-vocabulary situation. We expect this work will provoke a rethink of the current de facto semantic segmentation model design.


page 3

page 5

page 7

page 15

page 16

page 17


PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment

Despite the great progress made by deep CNNs in image semantic segmentat...

Visual Recognition with Deep Nearest Centroids

We devise deep nearest centroids (DNC), a conceptually elegant yet surpr...

Number-Adaptive Prototype Learning for 3D Point Cloud Semantic Segmentation

3D point cloud semantic segmentation is one of the fundamental tasks for...

Prototype Guided Network for Anomaly Segmentation

Semantic segmentation methods can not directly identify abnormal objects...

StructToken : Rethinking Semantic Segmentation with Structural Prior

In this paper, we present structure token (StructToken), a new paradigm ...

Fully Convolutional Open Set Segmentation

In semantic segmentation knowing about all existing classes is essential...

GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models

Prevalent semantic segmentation solutions are, in essence, a dense discr...

Please sign up or login with your details

Forgot password? Click here to reset