How to find a unicorn: a novel model-free, unsupervised anomaly detection method for time series
share
Recognition of anomalous events is a challenging but critical task in many scientific and industrial fields, especially when the properties of anomalies are unknown. In this paper, we present a new anomaly concept called "unicorn" or unique event and present a new, model-independent, unsupervised detection algorithm to detect unicorns. The Temporal Outlier Factor (TOF) is introduced to measure the uniqueness of events in continuous data sets from dynamic systems. The concept of unique events differ significantly from traditional outliers in many aspects: while repetitive outliers are no longer unique events, a unique event is not necessarily outlier in either pointwise or collective sense, it does not necessarily fall out from the distribution of normal activity. We examined the performance of our algorithm in recognizing unique events on different types of simulated data sets with anomalies and compared it with the standard Local Outlier Factor (LOF). TOF had superior performance compared to LOF even in recognizing traditional outliers and also recognized unique events that LOF did not. Benefits of the unicorn concept and the new detection method was illustrated by example data sets from very different scientific fields. In cases where the unique event is already known, our algorithm successfully recognized them: the gravitational waves of a black hole merger on LIGO detector data and the signs of respiratory failure on ECG data series. Furthermore, unique events were found on the LIBOR data set of the last 30 years.
READ FULL TEXT