Generalizing Importance Weighting to A Universal Solver for Distribution Shift Problems

by   Tongtong Fang, et al.

Distribution shift (DS) may have two levels: the distribution itself changes, and the support (i.e., the set where the probability density is non-zero) also changes. When considering the support change between the training and test distributions, there can be four cases: (i) they exactly match; (ii) the training support is wider (and thus covers the test support); (iii) the test support is wider; (iv) they partially overlap. Existing methods are good at cases (i) and (ii), while cases (iii) and (iv) are more common nowadays but still under-explored. In this paper, we generalize importance weighting (IW), a golden solver for cases (i) and (ii), to a universal solver for all cases. Specifically, we first investigate why IW may fail in cases (iii) and (iv); based on the findings, we propose generalized IW (GIW) that could handle cases (iii) and (iv) and would reduce to IW in cases (i) and (ii). In GIW, the test support is split into an in-training (IT) part and an out-of-training (OOT) part, and the expected risk is decomposed into a weighted classification term over the IT part and a standard classification term over the OOT part, which guarantees the risk consistency of GIW. Then, the implementation of GIW consists of three components: (a) the split of validation data is carried out by the one-class support vector machine, (b) the first term of the empirical risk can be handled by any IW algorithm given training data and IT validation data, and (c) the second term just involves OOT validation data. Experiments demonstrate that GIW is a universal solver for DS problems, outperforming IW methods in cases (iii) and (iv).


page 2

page 3

page 11

page 12

page 18

page 22

page 23


Effects of sampling skewness of the importance-weighted risk estimator on model selection

Importance-weighting is a popular and well-researched technique for deal...

On reducing sampling variance in covariate shift using control variates

Covariate shift classification problems can in principle be tackled by i...

Rethinking Importance Weighting for Deep Learning under Distribution Shift

Under distribution shift (DS) where the training data distribution diffe...

Machine Learning Based Radiomics for Glial Tumor Classification and Comparison with Volumetric Analysis

Purpose; The purpose of this study is to classify glial tumors into grad...

New Constructions of Complementary Sequence Pairs over 4^q-QAM

The previous constructions of quadrature amplitude modulation (QAM) Gola...

Weighted Risk Minimization & Deep Learning

Importance weighting is a key ingredient in many algorithms for causal i...

Phases of methodological research in biostatistics - building the evidence base for new methods

Although the biostatistical scientific literature publishes new methods ...

Please sign up or login with your details

Forgot password? Click here to reset