DeepAI AI Chat
Log In Sign Up

Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

by   Nicholas Larsen, et al.

The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.


page 1

page 2

page 3

page 4


Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring

With the growing needs of online A/B testing to support the innovation i...

Democratizing online controlled experiments at

There is an extensive literature about online controlled experiments, bo...

Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

Online controlled experiments are the primary tool for measuring the cau...

Online Aggregation based Approximate Query Processing: A Literature Survey

In the current world, OLAP (Online Analytical Processing) is used intens...

Datasets for Online Controlled Experiments

Online Controlled Experiments (OCE) are the gold standard to measure imp...

Covariance Estimation and its Application in Large-Scale Online Controlled Experiments

During the last few decades, online controlled experiments (also known a...

Internet of Everything Driven Neuromarketing: Key Technologies and Challenges

Preserving customers' expectations and understanding factors affecting t...