A Data Structure Perspective to the RDD-based Apriori Algorithm on Spark

by   Pankaj Singh, et al.

During the recent years, a number of efficient and scalable frequent itemset mining algorithms for big data analytics have been proposed by many researchers. Initially, MapReduce-based frequent itemset mining algorithms on Hadoop cluster were proposed. Although, Hadoop has been developed as a cluster computing system for handling and processing big data, but the performance of Hadoop does not meet the expectation for the iterative algorithms of data mining, due to its high I/O, and writing and then reading intermediate results in the disk. Consequently, Spark has been developed as another cluster computing infrastructure which is much faster than Hadoop due to its in-memory computation. It is highly suitable for iterative algorithms and supports batch, interactive, iterative, and stream processing of data. Many frequent itemset mining algorithms have been re-designed on the Spark, and most of them are Apriori-based. All these Spark-based Apriori algorithms use Hash Tree as the underlying data structure. This paper investigates the efficiency of various data structures for the Spark-based Apriori. Although, the data structure perspective has been investigated previously, but for MapReduce-based Apriori, and it must be re-investigated in the distributed computing environment of Spark. The considered underlying data structures are Hash Tree, Trie, and Hash Table Trie. The experimental results on the benchmark datasets show that the performance of Spark-based Apriori with Trie and Hash Table Trie are almost similar but both perform many times better than Hash Tree in the distributed computing environment of Spark.


page 1

page 2

page 3

page 4


RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework

Initially, a number of frequent itemset mining (FIM) algorithms have bee...

RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework (Extended Version)

Frequent itemset mining (FIM) is a highly computational and data intensi...

Scaling Shared-Memory Data Structures as Distributed Global-View Data Structures in the Partitioned Global Address Space model

The Partitioned Global Address Space (PGAS), a memory model in which the...

Analyzing Large-Scale, Distributed and Uncertain Data

The exponential growth of data in current times and the demand to gain i...

Performance Evaluation of Linear Regression Algorithm in Cluster Environment

Cluster computing was introduced to replace the superiority of super com...

A Self-Stabilizing Hashed Patricia Trie

While a lot of research in distributed computing has covered solutions f...

Parallel Algorithms for Adding a Collection of Sparse Matrices

We develop a family of parallel algorithms for the SpKAdd operation that...

Please sign up or login with your details

Forgot password? Click here to reset