Robustness Analysis on Foundational Segmentation Models

Due to the increase in computational resources and accessibility of data, an increase in large, deep learning models trained on copious amounts of data using self-supervised or semi-supervised learning have emerged. These "foundation" models are often adapted to a variety of downstream tasks like classification, object detection, and segmentation with little-to-no training on the target dataset. In this work, we perform a robustness analysis of Visual Foundation Models (VFMs) for segmentation tasks and compare them to supervised models of smaller scale. We focus on robustness against real-world distribution shift perturbations.We benchmark four state-of-the-art segmentation architectures using 2 different datasets, COCO and ADE20K, with 17 different perturbations with 5 severity levels each. We find interesting insights that include (1) VFMs are not robust to compression-based corruptions, (2) while the selected VFMs do not significantly outperform or exhibit more robustness compared to non-VFM models, they remain competitively robust in zero-shot evaluations, particularly when non-VFM are under supervision and (3) selected VFMs demonstrate greater resilience to specific categories of objects, likely due to their open-vocabulary training paradigm, a feature that non-VFM models typically lack. We posit that the suggested robustness evaluation introduces new requirements for foundational models, thus sparking further research to enhance their performance.


page 2

page 3

page 5

page 8

page 15

page 20

page 21

page 22


Masked Unsupervised Self-training for Zero-shot Image Classification

State-of-the-art computer vision models are mostly trained with supervis...

How Well Do Self-Supervised Models Transfer?

Self-supervised visual representation learning has seen huge progress in...

Empirical Analysis of a Segmentation Foundation Model in Prostate Imaging

Most state-of-the-art techniques for medical image segmentation rely on ...

Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models

When trained at a sufficient scale, self-supervised learning has exhibit...

Two Approaches to Supervised Image Segmentation

Though performed almost effortlessly by humans, segmenting 2D gray-scale...

Multi-modal Robustness Analysis Against Language and Visual Perturbations

Joint visual and language modeling on large-scale datasets has recently ...

Comparing Foundation Models using Data Kernels

Recent advances in self-supervised learning and neural network scaling h...

Please sign up or login with your details

Forgot password? Click here to reset