Towards Optimal Variance Reduction in Online Controlled Experiments

10/26/2021
by   Ying Jin, et al.
0

We study the optimal variance reduction solutions for online controlled experiments by applying flexible machine learning tools to incorporate covariates that are independent from the treatment but have predictive power for the outcomes. Employing cross-fitting, we propose variance reduction procedures for both count metrics and ratio metrics in online experiments based on which the inference of the estimands are valid under mild convergence conditions. We also establish the asymptotic optimality of all these procedures under consistency condition of the machine learning estimators. In complement to the proposed nonlinear optimal procedure, a linear adjustment method for ratio metrics is also derived as a special case that is computationally efficient and can flexibly incorporate any pre-treatment covariates. Comprehensive simulation studies are performed and practical suggestions are given. When tested on real online experiment data from LinkedIn, the proposed optimal procedure for ratio metrics can reduce up to 80% of variance compared to the standard difference-in-mean estimator and also further reduce up to 30% of variance compared to the CUPED approach by going beyond linearity and incorporating a large number of extra covariates.

READ FULL TEXT
research
06/14/2021

Machine Learning for Variance Reduction in Online Experiments

We consider the problem of variance reduction in randomized controlled t...
research
04/08/2019

Unbiased variance reduction in randomized experiments

This paper develops a flexible method for decreasing the variance of est...
research
09/23/2021

Blocking, rerandomization, and regression adjustment in randomized experiments with high-dimensional covariates

Blocking, a special case of rerandomization, is routinely implemented in...
research
09/11/2021

Variance Reduction for Causal Inference

Propensity score methods have been shown to be powerful in obtaining eff...
research
06/13/2018

Stratification Trees for Adaptive Randomization in Randomized Controlled Trials

This paper proposes a two-stage adaptive randomization procedure for ran...
research
05/31/2021

Regression-Adjusted Estimation of Quantile Treatment Effects under Covariate-Adaptive Randomizations

This paper examines regression-adjusted estimation and inference of unco...

Please sign up or login with your details

Forgot password? Click here to reset