GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models

by   Li Zaitang, et al.
The Chinese University of Hong Kong

Current studies on adversarial robustness mainly focus on aggregating local robustness results from a set of data samples to evaluate and rank different models. However, the local statistics may not well represent the true global robustness of the underlying unknown data distribution. To address this challenge, this paper makes the first attempt to present a new framework, called GREAT Score , for global robustness evaluation of adversarial perturbation using generative models. Formally, GREAT Score carries the physical meaning of a global statistic capturing a mean certified attack-proof perturbation level over all samples drawn from a generative model. For finite-sample evaluation, we also derive a probabilistic guarantee on the sample complexity and the difference between the sample mean and the true mean. GREAT Score has several advantages: (1) Robustness evaluations using GREAT Score are efficient and scalable to large models, by sparing the need of running adversarial attacks. In particular, we show high correlation and significantly reduced computation cost of GREAT Score when compared to the attack-based model ranking on RobustBench (Croce,et. al. 2021). (2) The use of generative models facilitates the approximation of the unknown data distribution. In our ablation study with different generative adversarial networks (GANs), we observe consistency between global robustness evaluation and the quality of GANs. (3) GREAT Score can be used for remote auditing of privacy-sensitive black-box models, as demonstrated by our robustness evaluation on several online facial recognition services.


Adversarial Robustness of Flow-Based Generative Models

Flow-based generative models leverage invertible generator functions to ...

DiffFlow: A Unified SDE Framework for Score-Based Diffusion Models and Generative Adversarial Networks

Generative models can be categorized into two types: explicit generative...

DaST: Data-free Substitute Training for Adversarial Attacks

Machine learning models are vulnerable to adversarial examples. For the ...

Generative Adversarial Networks

We propose a new framework for estimating generative models via an adver...

Understanding the Intrinsic Robustness of Image Distributions using Conditional Generative Models

Starting with Gilmer et al. (2018), several works have demonstrated the ...

Generate High-Resolution Adversarial Samples by Identifying Effective Features

As the prevalence of deep learning in computer vision, adversarial sampl...

Revisiting Classifier Two-Sample Tests

The goal of two-sample tests is to assess whether two samples, S_P ∼ P^n...

Please sign up or login with your details

Forgot password? Click here to reset