Panoramic Panoptic Segmentation: Towards Complete Surrounding Understanding via Unsupervised Contrastive Learning

by   Alexander Jaus, et al.

In this work, we introduce panoramic panoptic segmentation as the most holistic scene understanding both in terms of field of view and image level understanding. A complete surrounding understanding provides a maximum of information to the agent, which is essential for any intelligent vehicle in order to make informed decisions in a safety-critical dynamic environment such as real-world traffic. In order to overcome the lack of annotated panoramic images, we propose a framework which allows model training on standard pinhole images and transfers the learned features to a different domain. Using our proposed method, we manage to achieve significant improvements of over 5% measured in PQ over non-adapted models on our Wild Panoramic Panoptic Segmentation (WildPPS) dataset. We show that our proposed Panoramic Robust Feature (PRF) framework is not only suitable to improve performance on panoramic images but can be beneficial whenever model training and deployment are executed on data taken from different distributions. As an additional contribution, we publish WildPPS: The first panoramic panoptic image dataset to foster progress in surrounding perception.


page 1

page 3

page 7


Attention-based Neural Network for Driving Environment Complexity Perception

Environment perception is crucial for autonomous vehicle (AV) safety. Mo...

PanoContext-Former: Panoramic Total Scene Understanding with a Transformer

Panoramic image enables deeper understanding and more holistic perceptio...

Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras

Understanding the surrounding environment of the vehicle is still one of...

DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation through SwaftNet for Surrounding Sensing

Semantically interpreting the traffic scene is crucial for autonomous tr...

Capturing Omni-Range Context for Omnidirectional Segmentation

Convolutional Networks (ConvNets) excel at semantic segmentation and hav...

Towards Embodied Scene Description

Embodiment is an important characteristic for all intelligent agents (cr...

Please sign up or login with your details

Forgot password? Click here to reset