Scalable and Reliable Multi-Dimensional Aggregation of Sensor Data Streams

by   Sören Henning, et al.

Ever-increasing amounts of data and requirements to process them in real time lead to more and more analytics platforms and software systems being designed according to the concept of stream processing. A common area of application is the processing of continuous data streams from sensors, for example, IoT devices or performance monitoring tools. In addition to analyzing pure sensor data, analyses of data for groups of sensors often need to be performed as well. Therefore, data streams of the individual sensors have to be continuously aggregated to a data stream for a group. Motivated by a real-world application scenario, we propose that such a stream aggregation approach has to allow for aggregating sensors in hierarchical groups, support multiple such hierarchies in parallel, provide reconfiguration at runtime, and preserve the scalability and reliability qualities induced by applying stream processing techniques. We propose a stream processing architecture fulfilling these requirements, which can be integrated into existing big data architectures. We present a pilot implementation of such an extended architecture and show how it is used in industry. Furthermore, in experimental evaluations we show that our solution scales linearly with the amount of sensors and provides adequate reliability in the case of faults.


Data Stream Clustering: A Review

Number of connected devices is steadily increasing and these devices con...

SenseRT: A Streaming Architecture for Smart Building Sensors

Building Management Systems (BMSs) have evolved in recent years, in ways...

Dependable IoT Data Stream Processing for Monitoring and Control of Urban Infrastructures

The Internet of Things describes a network of physical devices interacti...

DataX: A system for Data eXchange and transformation of streams

The exponential growth in smart sensors and rapid progress in 5G network...

A Scalable Architecture for Power Consumption Monitoring in Industrial Production Environments

Detailed knowledge about the electrical power consumption in industrial ...

RelSen: An Optimization-based Framework for Simultaneously Sensor Reliability Monitoring and Data Cleaning

Recent advances in the Internet of Things (IoT) technology have led to a...

A Scalable and Robust Framework for Data Stream Ingestion

An essential part of building a data-driven organization is the ability ...

Please sign up or login with your details

Forgot password? Click here to reset