DeepAI AI Chat
Log In Sign Up

Variance Reduction for Experiments with One-Sided Triggering using CUPED

by   Alex Deng, et al.
Airbnb, Inc.

In online experimentation, trigger-dilute analysis is an approach to obtain more precise estimates of intent-to-treat (ITT) effects when the intervention is only exposed, or "triggered", for a small subset of the population. Trigger-dilute analysis cannot be used for estimation when triggering is only partially observed. In this paper, we propose an unbiased ITT estimator with reduced variance for cases where triggering status is only observed in the treatment group. Our method is based on the efficiency augmentation idea of CUPED and draws upon identification frameworks from the principal stratification and instrumental variables literature. The unbiasedness of our estimation approach relies on a testable assumption that an augmentation term used for covariate adjustment equals zero in expectation. When this augmentation term fails a mean-zero test, we show how our estimator can incorporate in-experiment observations to reduce the augmentation's bias, by sacrificing the amount of variance reduced. This provides an explicit knob to trade off bias with variance. We demonstrate through simulations that our estimator can remain unbiased and achieve precision improvements as good as if triggering status were fully observed, and in some cases outperforms trigger-dilute analysis.


page 1

page 2

page 3

page 4


Designing Transportable Experiments

We consider the problem of designing a randomized experiment on a source...

Improving instrumental variable estimators with post-stratification

An instrumental variable (IV) is a device that encourages units in a stu...

An unbiased ray-marching transmittance estimator

We present an in-depth analysis of the sources of variance in state-of-t...

Asymptotics of Cointegration Estimator with Misspecified Rank

Cointegration analysis was developed for non-stationary linear processes...

Unbiased Estimation of the Hessian for Partially Observed Diffusions

In this article we consider the development of unbiased estimators of th...

Entropy Regularization for Population Estimation

Entropy regularization is known to improve exploration in sequential dec...

Interference, Bias, and Variance in Two-Sided Marketplace Experimentation: Guidance for Platforms

Two-sided marketplace platforms often run experiments to test the effect...