An Offline Metric for the Debiasedness of Click Models

by   Romain Deffayet, et al.

A well-known problem when learning from user clicks are inherent biases prevalent in the data, such as position or trust bias. Click models are a common method for extracting information from user clicks, such as document relevance in web search, or to estimate click biases for downstream applications such as counterfactual learning-to-rank, ad placement, or fair ranking. Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks in which the ranking distribution differs from the training distribution, i.e., under covariate shift. In this work, we propose an evaluation metric based on conditional independence testing to detect a lack of robustness to covariate shift in click models. We introduce the concept of debiasedness and a metric for measuring it. We prove that debiasedness is a necessary condition for recovering unbiased and consistent relevance scores and for the invariance of click prediction under covariate shift. In extensive semi-synthetic experiments, we show that our proposed metric helps to predict the downstream performance of click models under covariate shift and is useful in an off-policy model selection setting.


page 1

page 2

page 3

page 4


Off-Policy Evaluation and Learning for External Validity under a Covariate Shift

We consider the evaluation and training of a new policy for the evaluati...

NICO++: Towards Better Benchmarking for Domain Generalization

Despite the remarkable performance that modern deep neural networks have...

Automatic Debiased Machine Learning for Covariate Shifts

In this paper we address the problem of bias in machine learning of para...

Federated Covariate Shift Adaptation for Missing Target Output Values

The most recent multi-source covariate shift algorithm is an efficient h...

Fair Classification under Covariate Shift and Missing Protected Attribute – an Investigation using Related Features

This study investigated the problem of fair classification under Covaria...

Contrastive Identification of Covariate Shift in Image Data

Identifying covariate shift is crucial for making machine learning syste...

On the Use of Sparse Filtering for Covariate Shift Adaptation

In this paper we formally analyse the use of sparse filtering algorithms...

Please sign up or login with your details

Forgot password? Click here to reset