Approximated Doubly Robust Search Relevance Estimation

by   Lixin Zou, et al.

Extracting query-document relevance from the sparse, biased clickthrough log is among the most fundamental tasks in the web search system. Prior art mainly learns a relevance judgment model with semantic features of the query and document and ignores directly counterfactual relevance evaluation from the clicking log. Though the learned semantic matching models can provide relevance signals for tail queries as long as the semantic feature is available. However, such a paradigm lacks the capability to introspectively adjust the biased relevance estimation whenever it conflicts with massive implicit user feedback. The counterfactual evaluation methods, on the contrary, ensure unbiased relevance estimation with sufficient click information. However, they suffer from the sparse or even missing clicks caused by the long-tailed query distribution. In this paper, we propose to unify the counterfactual evaluating and learning approaches for unbiased relevance estimation on search queries with various popularities. Specifically, we theoretically develop a doubly robust estimator with low bias and variance, which intentionally combines the benefits of existing relevance evaluating and learning approaches. We further instantiate the proposed unbiased relevance estimation framework in Baidu search, with comprehensive practical solutions designed regarding the data pipeline for click behavior tracking and online relevance estimation with an approximated deep neural network. Finally, we present extensive empirical evaluations to verify the effectiveness of our proposed framework, finding that it is robust in practice and manages to improve online ranking performance substantially.


page 1

page 2

page 3

page 4


Personalized Search Via Neural Contextual Semantic Relevance Ranking

Existing neural relevance models do not give enough consideration for qu...

Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank

Unbiased CLTR requires click propensities to compensate for the differen...

Counterfactual Learning to Rank using Heterogeneous Treatment Effect Estimation

Learning-to-Rank (LTR) models trained from implicit feedback (e.g. click...

Refining Recency Search Results with User Click Feedback

Traditional machine-learned ranking systems for web search are often tra...

Robust Generalization and Safe Query-Specialization in Counterfactual Learning to Rank

Existing work in counterfactual Learning to Rank (LTR) has focussed on o...

Estimating Position Bias without Intrusive Interventions

Presentation bias is one of the key challenges when learning from implic...

Model-based Unbiased Learning to Rank

Unbiased Learning to Rank (ULTR) that learns to rank documents with bias...

Please sign up or login with your details

Forgot password? Click here to reset