Statistical Challenges in Online Controlled Experiments: A Review of A/B Testing Methodology

12/21/2022
by   Nicholas Larsen, et al.
0

The rise of internet-based services and products in the late 1990's brought about an unprecedented opportunity for online businesses to engage in large scale data-driven decision making. Over the past two decades, organizations such as Airbnb, Alibaba, Amazon, Baidu, Booking, Alphabet's Google, LinkedIn, Lyft, Meta's Facebook, Microsoft, Netflix, Twitter, Uber, and Yandex have invested tremendous resources in online controlled experiments (OCEs) to assess the impact of innovation on their customers and businesses. Running OCEs at scale has presented a host of challenges requiring solutions from many domains. In this paper we review challenges that require new statistical methodologies to address them. In particular, we discuss the practice and culture of online experimentation, as well as its statistics literature, placing the current methodologies within their relevant statistical lineages and providing illustrative examples of OCE applications. Our goal is to raise academic statisticians' awareness of these new research opportunities to increase collaboration between academia and the online industry.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/02/2023

Experimentation Platforms Meet Reinforcement Learning: Bayesian Sequential Decision-Making for Continuous Monitoring

With the growing needs of online A/B testing to support the innovation i...
research
10/23/2017

Democratizing online controlled experiments at Booking.com

There is an extensive literature about online controlled experiments, bo...
research
03/16/2018

Online Controlled Experiments for Personalised e-Commerce Strategies: Design, Challenges, and Pitfalls

Online controlled experiments are the primary tool for measuring the cau...
research
04/14/2022

Online Aggregation based Approximate Query Processing: A Literature Survey

In the current world, OLAP (Online Analytical Processing) is used intens...
research
11/19/2021

Datasets for Online Controlled Experiments

Online Controlled Experiments (OCE) are the gold standard to measure imp...
research
08/05/2021

Covariance Estimation and its Application in Large-Scale Online Controlled Experiments

During the last few decades, online controlled experiments (also known a...
research
05/30/2021

Internet of Everything Driven Neuromarketing: Key Technologies and Challenges

Preserving customers' expectations and understanding factors affecting t...

Please sign up or login with your details

Forgot password? Click here to reset