Grand Challenge: Optimized Stage Processing for Anomaly Detection on Numerical Data Streams
share
The 2017 Grand Challenge focused on the problem of automatic detection of anomalies for manufacturing equipment. This paper reports the technical details of a solution focused on particular optimizations of the processing stages. These included customized input parsing, fine tuning of a k-means clustering algorithm and probability analysis using a lazy flavor of a Markov chain. We have observed in our custom implementation that carefully tweaking these processing stages at single node level by leveraging various data stream characteristics can yield good performance results. We start the paper with several observations concerning the input data stream, following with our solution description with details on particular optimizations, and we conclude with evaluation and a discussion of obtained results.
READ FULL TEXT