GreedyGD: Enhanced Generalized Deduplication for Direct Analytics in IoT

by   Aaron Hurst, et al.

Exponential growth in the amount of data generated by the Internet of Things currently pose significant challenges for data communication, storage and analytics and leads to high costs for organisations hoping to leverage their data. Novel techniques are therefore needed to holistically improve the efficiency of data storage and analytics in IoT systems. The emerging compression technique Generalized Deduplication (GD) has been shown to deliver high compression and enable direct compressed data analytics with low storage and memory requirements. In this paper, we propose a new GD-based data compression algorithm called GreedyGD that is designed for analytics. Compared to existing versions of GD, GreedyGD enables more reliable analytics with less data, while running 11.2x faster and delivering even better compression.


page 1

page 2

page 3

page 4


TADOC: Text Analytics Directly on Compression

This article provides a comprehensive description of Text Analytics Dire...

Change a Bit to save Bytes: Compression for Floating Point Time-Series Data

The number of IoT devices is expected to continue its dramatic growth in...

A milestone for FaaS pipelines; object storage vs VM-driven data exchange

Serverless functions provide high levels of parallelism, short startup t...

Slim Graph: Practical Lossy Graph Compression for Approximate Graph Processing, Storage, and Analytics

We propose Slim Graph: the first programming model and framework for pra...

Revisit Visual Representation in Analytics Taxonomy: A Compression Perspective

Visual analytics have played an increasingly critical role in the Intern...

An Empirical Evaluation of Columnar Storage Formats

Columnar storage is one of the core components of a modern data analytic...

Leveraging Domain Knowledge using Machine Learning for Image Compression in Internet-of-Things

The emergent ecosystems of intelligent edge devices in diverse Internet ...

Please sign up or login with your details

Forgot password? Click here to reset