RobustBench: a standardized adversarial robustness benchmark

10/19/2020
by   Francesco Croce, et al.
4

Evaluation of adversarial robustness is often error-prone leading to overestimation of the true robustness of models. While adaptive attacks designed for a particular defense are a way out of this, there are only approximate guidelines on how to perform them. Moreover, adaptive evaluations are highly customized for particular models, which makes it difficult to compare different defenses. Our goal is to establish a standardized benchmark of adversarial robustness, which as accurately as possible reflects the robustness of the considered models within a reasonable computational budget. This requires to impose some restrictions on the admitted models to rule out defenses that only make gradient-based attacks ineffective without improving actual robustness. We evaluate robustness of models for our benchmark with AutoAttack, an ensemble of white- and black-box attacks which was recently shown in a large-scale study to improve almost all robustness evaluations compared to the original publications. Our leaderboard, hosted at http://robustbench.github.io/, aims at reflecting the current state of the art on a set of well-defined tasks in ℓ_∞- and ℓ_2-threat models with possible extensions in the future. Additionally, we open-source the library http://github.com/RobustBench/robustbench that provides unified access to state-of-the-art robust models to facilitate their downstream applications. Finally, based on the collected models, we analyze general trends in ℓ_p-robustness and its impact on other tasks such as robustness to various distribution shifts and out-of-distribution detection.

READ FULL TEXT

page 1

page 2

page 3

page 4

06/18/2021

Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples

Evaluating robustness of machine-learning models to adversarial examples...
06/18/2020

Beware the Black-Box: on the Robustness of Recent Defenses to Adversarial Examples

Recent defenses published at venues like NIPS, ICML, ICLR and CVPR are m...
11/15/2022

MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack

Adversarial attacks can deceive neural networks by adding tiny perturbat...
06/14/2023

On the Robustness of Latent Diffusion Models

Latent diffusion models achieve state-of-the-art performance on a variet...
11/08/2021

Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning

Adversarial attacks on graphs have posed a major threat to the robustnes...
02/22/2023

On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

ChatGPT is a recent chatbot service released by OpenAI and is receiving ...

Please sign up or login with your details

Forgot password? Click here to reset