Hunter: Using Change Point Detection to Hunt for Performance Regressions

by   Matt Fleming, et al.

Change point detection has recently gained popularity as a method of detecting performance changes in software due to its ability to cope with noisy data. In this paper we present Hunter, an open source tool that automatically detects performance regressions and improvements in time-series data. Hunter uses a modified E-divisive means algorithm to identify statistically significant changes in normally-distributed performance metrics. We describe the changes we made to the E-divisive means algorithm along with their motivation. The main change we adopted was to replace the significance test using randomized permutations with a Student's t-test, as we discovered that the randomized approach did not produce deterministic results, at least not with a reasonable number of iterations. In addition we've made tweaks that allow us to find change points the original algorithm would not, such as two nearby changes. For evaluation, we developed a method to generate real timeseries, but with artificially injected changes in latency. We used these data sets to compare Hunter against two other well known algorithms, PELT and DYNP. Finally, we conclude with lessons we've learned supporting Hunter across teams with individual responsibility for the performance of their project.


page 1

page 2

page 3

page 4


Change Point Detection in Software Performance Testing

We describe our process for automatic detection of performance changes f...

Bayesian Online Change Point Detection for Baseline Shifts

In time series data analysis, detecting change points on a real-time bas...

Detecting Multiple Step Changes Using Adaptive Regression Splines with Application to Neural Recordings

Time series produced by dynamical systems as frequently the case in neur...

Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation

Change point detection (CPD) aims to locate abrupt property changes in t...

Automatic Change-Point Detection in Time Series via Deep Learning

Detecting change-points in data is challenging because of the range of p...

A Review of Changepoint Detection Models

The objective of the change-point detection is to discover the abrupt pr...

Detecting Change Intervals with Isolation Distributional Kernel

Detecting abrupt changes in data distribution is one of the most signifi...

Please sign up or login with your details

Forgot password? Click here to reset