False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments

by   Yuxiang Xie, et al.

Online controlled experiments (a.k.a. A/B testing) have been used as the mantra for data-driven decision making on feature changing and product shipping in many Internet companies. However, it is still a great challenge to systematically measure how every code or feature change impacts millions of users with great heterogeneity (e.g. countries, ages, devices). The most commonly used A/B testing framework in many companies is based on Average Treatment Effect (ATE), which cannot detect the heterogeneity of treatment effect on users with different characteristics. In this paper, we propose statistical methods that can systematically and accurately identify Heterogeneous Treatment Effect (HTE) of any user cohort of interest (e.g. mobile device type, country), and determine which factors (e.g. age, gender) of users contribute to the heterogeneity of the treatment effect in an A/B test. By applying these methods on both simulation data and real-world experimentation data, we show how they work robustly with controlled low False Discover Rate (FDR), and at the same time, provides us with useful insights about the heterogeneity of identified user groups. We have deployed a toolkit based on these methods, and have used it to measure the Heterogeneous Treatment Effect of many A/B tests at Snap.


Treatment Effect Detection with Controlled FDR under Dependence for Large-Scale Experiments

Online controlled experiments (also known as A/B Testing) have been view...

Personalization and Optimization of Decision Parameters via Heterogenous Causal Effects

Randomized experimentation (also known as A/B testing or bucket testing)...

Inform Product Change through Experimentation with Data-Driven Behavioral Segmentation

Online controlled experimentation is widely adopted for evaluating new f...

LinkLouvain: Link-Aware A/B Testing and Its Application on Online Marketing Campaign

A lot of online marketing campaigns aim to promote user interaction. The...

An evaluation framework for personalization strategy experiment designs

Online Controlled Experiments (OCEs) are the gold standard in evaluating...

Trustworthy Experimentation Under Telemetry Loss

Failure to accurately measure the outcomes of an experiment can lead to ...

Please sign up or login with your details

Forgot password? Click here to reset