A Domain-Region Based Evaluation of ML Performance Robustness to Covariate Shift

by   Firas Bayram, et al.

Most machine learning methods assume that the input data distribution is the same in the training and testing phases. However, in practice, this stationarity is usually not met and the distribution of inputs differs, leading to unexpected performance of the learned model in deployment. The issue in which the training and test data inputs follow different probability distributions while the input-output relationship remains unchanged is referred to as covariate shift. In this paper, the performance of conventional machine learning models was experimentally evaluated in the presence of covariate shift. Furthermore, a region-based evaluation was performed by decomposing the domain of probability density function of the input data to assess the classifier's performance per domain region. Distributional changes were simulated in a two-dimensional classification problem. Subsequently, a higher four-dimensional experiments were conducted. Based on the experimental analysis, the Random Forests algorithm is the most robust classifier in the two-dimensional case, showing the lowest degradation rate for accuracy and F1-score metrics, with a range between 0.1 reveal that in higher-dimensional experiments, the performance of the models is predominantly influenced by the complexity of the classification function, leading to degradation rates exceeding 25 that the models exhibit high bias towards the region with high density in the input space domain of the training samples.


page 5

page 17


Information Geometrically Generalized Covariate Shift Adaptation

Many machine learning methods assume that the training and test data fol...

Covariate Shift in High-Dimensional Random Feature Regression

A significant obstacle in the development of robust machine learning mod...

Robust Covariate Shift Prediction with General Losses and Feature Views

Covariate shift relaxes the widely-employed independent and identically ...

Adapting to Continuous Covariate Shift via Online Density Ratio Estimation

Dealing with distribution shifts is one of the central challenges for mo...

Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria

Covariate distribution shifts and adversarial perturbations present robu...

A Distribution-Free Test of Covariate Shift Using Conformal Prediction

Covariate shift is a common and important assumption in transfer learnin...

Robust Correction of Sampling Bias Using Cumulative Distribution Functions

Varying domains and biased datasets can lead to differences between the ...

Please sign up or login with your details

Forgot password? Click here to reset