Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

03/28/2019
by   Dan Hendrycks, et al.
berkeley college
Oregon State University
30

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.

READ FULL TEXT

page 3

page 4

page 8

page 12

07/04/2018

Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations

In this paper we establish rigorous benchmarks for image classifier robu...
03/02/2021

Benchmarking Robustness of Deep Learning Classifiers Using Two-Factor Perturbation

This paper adds to the fundamental body of work on benchmarking the robu...
09/18/2023

Evaluating Adversarial Robustness with Expected Viable Performance

We introduce a metric for evaluating the robustness of a classifier, wit...
09/02/2021

Real World Robustness from Systematic Noise

Systematic error, which is not determined by chance, often refers to the...
04/02/2021

Defending Against Image Corruptions Through Adversarial Augmentations

Modern neural networks excel at image classification, yet they remain vu...
03/30/2023

ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing

Recent studies have shown that higher accuracy on ImageNet usually leads...
06/05/2019

A systematic framework for natural perturbations from videos

We introduce a systematic framework for quantifying the robustness of cl...

Code Repositories

ttt_imagenet_release

TTT Code Release


view repo

ttt_cifar_release

TTT Code Release


view repo

datasets

Download scripts to open datasets.


view repo

Please sign up or login with your details

Forgot password? Click here to reset