Improving Fast Adversarial Training with Prior-Guided Knowledge

04/01/2023
by   Xiaojun Jia, et al.
University of the Chinese Academy of Sciences
iie.ac.cn
Beihang University
The Chinese University of Hong Kong, Shenzhen
SUN YAT-SEN UNIVERSITY
0

Fast adversarial training (FAT) is an efficient method to improve robustness. However, the original FAT suffers from catastrophic overfitting, which dramatically and suddenly reduces robustness after a few training epochs. Although various FAT variants have been proposed to prevent overfitting, they require high training costs. In this paper, we investigate the relationship between adversarial example quality and catastrophic overfitting by comparing the training processes of standard adversarial training and FAT. We find that catastrophic overfitting occurs when the attack success rate of adversarial examples becomes worse. Based on this observation, we propose a positive prior-guided adversarial initialization to prevent overfitting by improving adversarial example quality without extra training costs. This initialization is generated by using high-quality adversarial perturbations from the historical training process. We provide theoretical analysis for the proposed initialization and propose a prior-guided regularization method that boosts the smoothness of the loss function. Additionally, we design a prior-guided ensemble FAT method that averages the different model weights of historical models using different decay rates. Our proposed method, called FGSM-PGK, assembles the prior-guided knowledge, i.e., the prior-guided initialization and model weights, acquired during the historical training process. Evaluations of four datasets demonstrate the superiority of the proposed method.

READ FULL TEXT

page 12

page 14

07/18/2022

Prior-Guided Adversarial Initialization for Fast Adversarial Training

Fast adversarial training (FAT) effectively improves the efficiency of s...
11/21/2021

Local Linearity and Double Descent in Catastrophic Overfitting

Catastrophic overfitting is a phenomenon observed during Adversarial Tra...
08/24/2023

Fast Adversarial Training with Smooth Convergence

Fast adversarial training (FAT) is beneficial for improving the adversar...
08/25/2021

Bridged Adversarial Training

Adversarial robustness is considered as a required property of deep neur...
10/11/2021

Boosting Fast Adversarial Training with Learnable Adversarial Initialization

Adversarial training (AT) has been demonstrated to be effective in impro...

Please sign up or login with your details

Forgot password? Click here to reset