A Method for Measuring Network Effects of One-to-One Communication Features in Online A/B Tests

by   Guillaume Saint-Jacques, et al.

A/B testing is an important decision making tool in product development because can provide an accurate estimate of the average treatment effect of a new features, which allows developers to understand how the business impact of new changes to products or algorithms. However, an important assumption of A/B testing, Stable Unit Treatment Value Assumption (SUTVA), is not always a valid assumption to make, especially for products that facilitate interactions between individuals. In contexts like one-to-one messaging we should expect network interference; if an experimental manipulation is effective, behavior of the treatment group is likely to influence members in the control group by sending them messages, violating this assumption. In this paper, we propose a novel method that can be used to account for network effects when A/B testing changes to one-to-one interactions. Our method is an edge-based analysis that can be applied to standard Bernoulli randomized experiments to retrieve an average treatment effect that is not influenced by network interference. We develop a theoretical model, and methods for computing point estimates and variances of effects of interest via network-consistent permutation testing. We then apply our technique to real data from experiments conducted on the messaging product at LinkedIn. We find empirical support for our model, and evidence that the standard method of analysis for A/B tests underestimates the impact of new features in one-to-one messaging contexts.


page 1

page 2

page 3

page 4


Cluster-Adaptive Network A/B Testing: From Randomization to Estimation

A/B testing is an important decision-making tool in product development ...

LinkLouvain: Link-Aware A/B Testing and Its Application on Online Marketing Campaign

A lot of online marketing campaigns aim to promote user interaction. The...

Detecting Interference in A/B Testing with Increasing Allocation

In the past decade, the technology industry has adopted online randomize...

Measuring the Demand Effects of Formal and Informal Communication : Evidence from Online Markets for Illicit Drugs

I present evidence that communication between marketplace participants i...

Design-Based Confidence Sequences for Anytime-valid Causal Inference

Many organizations run thousands of randomized experiments, or A/B tests...

A Study of Symbiosis Bias in A/B Tests of Recommendation Algorithms

One assumption underlying the unbiasedness of global treatment effect es...

Using Ego-Clusters to Measure Network Effects at LinkedIn

A network effect is said to take place when a new feature not only impac...

Please sign up or login with your details

Forgot password? Click here to reset