Real Negatives Matter: Continuous Training with Real Negatives for Delayed Feedback Modeling

by   Siyu Gu, et al.

One of the difficulties of conversion rate (CVR) prediction is that the conversions can delay and take place long after the clicks. The delayed feedback poses a challenge: fresh data are beneficial to continuous training but may not have complete label information at the time they are ingested into the training pipeline. To balance model freshness and label certainty, previous methods set a short waiting window or even do not wait for the conversion signal. If conversion happens outside the waiting window, this sample will be duplicated and ingested into the training pipeline with a positive label. However, these methods have some issues. First, they assume the observed feature distribution remains the same as the actual distribution. But this assumption does not hold due to the ingestion of duplicated samples. Second, the certainty of the conversion action only comes from the positives. But the positives are scarce as conversions are sparse in commercial systems. These issues induce bias during the modeling of delayed feedback. In this paper, we propose DElayed FEedback modeling with Real negatives (DEFER) method to address these issues. The proposed method ingests real negative samples into the training pipeline. The ingestion of real negatives ensures the observed feature distribution is equivalent to the actual distribution, thus reducing the bias. The ingestion of real negatives also brings more certainty information of the conversion. To correct the distribution shift, DEFER employs importance sampling to weigh the loss function. Experimental results on industrial datasets validate the superiority of DEFER. DEFER have been deployed in the display advertising system of Alibaba, obtaining over 6.0 in several scenarios. The code and data in this paper are now open-sourced


page 1

page 2

page 3

page 4


Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction

Alleviating the delayed feedback problem is of crucial importance for th...

Freshness or Accuracy, Why Not Both? Addressing Delayed Feedback via Dynamic Graph Neural Networks

The delayed feedback problem is one of the most pressing challenges in p...

Learning Classifiers under Delayed Feedback with a Time Window Assumption

We consider training a binary classifier under delayed feedback (DF Lear...

Dual Learning Algorithm for Delayed Feedback in Display Advertising

In display advertising, predicting the conversion rate, that is, the pro...

Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR prediction

One of the challenges in display advertising is that the distribution of...

A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback

In display advertising, predicting the conversion rate, that is, the pro...

A Nonparametric Delayed Feedback Model for Conversion Rate Prediction

Predicting conversion rates (CVRs) in display advertising (e.g., predict...

Please sign up or login with your details

Forgot password? Click here to reset