Statistical Learning under Heterogenous Distribution Shift

by   Max Simchowitz, et al.

This paper studies the prediction of a target 𝐳 from a pair of random variables (𝐱,𝐲), where the ground-truth predictor is additive 𝔼[𝐳|𝐱,𝐲] = f_⋆(𝐱) +g_⋆(𝐲). We study the performance of empirical risk minimization (ERM) over functions f+g, f βˆˆβ„± and g βˆˆπ’’, fit on a given training distribution, but evaluated on a test distribution which exhibits covariate shift. We show that, when the class β„± is "simpler" than 𝒒 (measured, e.g., in terms of its metric entropy), our predictor is more resilient to heterogenous covariate shifts in which the shift in 𝐱 is much greater than that in 𝐲. These results rely on a novel HΓΆlder style inequality for the Dudley integral which may be of independent interest. Moreover, we corroborate our theoretical findings with experiments demonstrating improved resilience to shifts in "simpler" features across numerous domains.

βˆ™ 02/06/2023

Adapting to Continuous Covariate Shift via Online Density Ratio Estimation

Dealing with distribution shifts is one of the central challenges for mo...
βˆ™ 12/05/2022

Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria

Covariate distribution shifts and adversarial perturbations present robu...
βˆ™ 12/21/2022

Adapting to Latent Subgroup Shifts via Concepts and Proxies

We address the problem of unsupervised domain adaptation when the source...
βˆ™ 09/17/2022

Mitigating Both Covariate and Conditional Shift for Domain Generalization

Domain generalization (DG) aims to learn a model on several source domai...
βˆ™ 07/11/2023

On the Need for a Language Describing Distribution Shifts: Illustrations on Tabular Datasets

Different distribution shifts require different algorithmic and operatio...
βˆ™ 06/17/2021

PAC Prediction Sets Under Covariate Shift

An important challenge facing modern machine learning is how to rigorous...
βˆ™ 06/06/2022

Tackling covariate shift with node-based Bayesian neural networks

Bayesian neural networks (BNNs) promise improved generalization under co...

Please sign up or login with your details

Forgot password? Click here to reset