Holistic Segmentation

by   Stefano Gasperini, et al.

As panoptic segmentation provides a prediction for every pixel in input, non-standard and unseen objects systematically lead to wrong outputs. However, in safety-critical settings, robustness against out-of-distribution samples and corner cases is crucial to avoid dangerous behaviors, such as ignoring an animal or a lost cargo on the road. Since driving datasets cannot contain enough data points to properly sample the long tail of the underlying distribution, a method must deal with unknown and unseen scenarios to be deployed safely. Previous methods targeted part of this issue, by re-identifying already seen unlabeled objects. In this work, we broaden the scope proposing holistic segmentation: a task to identify and separate unseen unknown objects into instances, without learning from unknowns, while performing panoptic segmentation of known classes. We tackle this new problem with U3HS, which first finds unknowns as highly uncertain regions, then clusters the corresponding instance-aware embeddings into individual objects. By doing so, for the first time in panoptic segmentation with unknown objects, our U3HS is not trained with unknown data, thus leaving the settings unconstrained with respect to the type of objects and allowing for a holistic scene understanding. Extensive experiments and comparisons on two public datasets, namely Cityscapes and Lost Found as a transfer, demonstrate the effectiveness of U3HS in the challenging task of holistic segmentation, with competitive closed-set panoptic segmentation performance.


page 1

page 9

page 12

page 13


SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation

State-of-the-art semantic or instance segmentation deep neural networks ...

Identifying Unknown Instances for Autonomous Driving

In the past few years, we have seen great progress in perception algorit...

Video Class Agnostic Segmentation Benchmark for Autonomous Driving

Semantic segmentation approaches are typically trained on large-scale da...

GOSS: Towards Generalized Open-set Semantic Segmentation

In this paper, we present and study a new image segmentation task, calle...

UGainS: Uncertainty Guided Anomaly Instance Segmentation

A single unexpected object on the road can cause an accident or may lead...

Learning to Better Segment Objects from Unseen Classes with Unlabeled Videos

The ability to localize and segment objects from unseen classes would op...

Detecting the Unexpected via Image Resynthesis

Classical semantic segmentation methods, including the recent deep learn...

Please sign up or login with your details

Forgot password? Click here to reset