Global-Local Regularization Via Distributional Robustness

by   Hoang Phan, et al.

Despite superior performance in many situations, deep neural networks are often vulnerable to adversarial examples and distribution shifts, limiting model generalization ability in real-world applications. To alleviate these problems, recent approaches leverage distributional robustness optimization (DRO) to find the most challenging distribution, and then minimize loss function over this most challenging distribution. Regardless of achieving some improvements, these DRO approaches have some obvious limitations. First, they purely focus on local regularization to strengthen model robustness, missing a global regularization effect which is useful in many real-world applications (e.g., domain adaptation, domain generalization, and adversarial machine learning). Second, the loss functions in the existing DRO approaches operate in only the most challenging distribution, hence decouple with the original distribution, leading to a restrictive modeling capability. In this paper, we propose a novel regularization technique, following the veins of Wasserstein-based DRO framework. Specifically, we define a particular joint distribution and Wasserstein-based uncertainty, allowing us to couple the original and most challenging distributions for enhancing modeling capability and applying both local and global regularizations. Empirical studies on different learning problems demonstrate that our proposed approach significantly outperforms the existing regularization approaches in various domains: semi-supervised learning, domain adaptation, domain generalization, and adversarial machine learning.


page 1

page 2

page 3

page 4


Unsupervised Domain Adaptation with Adversarial Residual Transform Networks

Domain adaptation is widely used in learning problems lacking labels. Re...

Adversarial Robustness for Unsupervised Domain Adaptation

Extensive Unsupervised Domain Adaptation (UDA) studies have shown great ...

Semi-supervised representation learning via dual autoencoders for domain adaptation

Domain adaptation which pays attention to exploiting the knowledge in so...

DEJA VU: Continual Model Generalization For Unseen Domains

In real-world applications, deep learning models often run in non-statio...

Generalizability of Adversarial Robustness Under Distribution Shifts

Recent progress in empirical and certified robustness promises to delive...

Certifying Out-of-Domain Generalization for Blackbox Functions

Certifying the robustness of model performance under bounded data distri...

Optimal Transport Model Distributional Robustness

Distributional robustness is a promising framework for training deep lea...

Please sign up or login with your details

Forgot password? Click here to reset