Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization

by   Xilie Xu, et al.

Adversarial contrastive learning (ACL), without requiring labels, incorporates adversarial data with standard contrastive learning (SCL) and outputs a robust representation which is generalizable and resistant to adversarial attacks and common corruptions. The style-independence property of representations has been validated to be beneficial in improving robustness transferability. Standard invariant regularization (SIR) has been proposed to make the learned representations via SCL to be independent of the style factors. However, how to equip robust representations learned via ACL with the style-independence property is still unclear so far. To this end, we leverage the technique of causal reasoning to propose an adversarial invariant regularization (AIR) that enforces robust representations learned via ACL to be style-independent. Then, we enhance ACL using invariant regularization (IR), which is a weighted sum of SIR and AIR. Theoretically, we show that AIR implicitly encourages the prediction of adversarial data and consistency between adversarial and natural data to be independent of data augmentations. We also theoretically demonstrate that the style-independence property of robust representation learned via ACL still holds in downstream tasks, providing generalization guarantees. Empirically, our comprehensive experimental results corroborate that IR can significantly improve the performance of ACL and its variants on various datasets.


page 1

page 2

page 3

page 4


Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection

Adversarial contrastive learning (ACL) does not require expensive data a...

Generalization Bounds for Adversarial Contrastive Learning

Deep networks are well-known to be fragile to adversarial attacks, and a...

Adversarial Cross-View Disentangled Graph Contrastive Learning

Graph contrastive learning (GCL) is prevalent to tackle the supervision ...

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Contrastive learning (CL) can learn generalizable feature representation...

Understanding and Achieving Efficient Robustness with Adversarial Contrastive Learning

Contrastive learning (CL) has recently emerged as an effective approach ...

Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective

Learning behavioral patterns from observational data has been a de-facto...

Please sign up or login with your details

Forgot password? Click here to reset