RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework (Extended Version)

by   Pankaj Singh, et al.

Frequent itemset mining (FIM) is a highly computational and data intensive algorithm. Therefore, parallel and distributed FIM algorithms have been designed to process large volume of data in a reduced time. Recently, a number of FIM algorithms have been designed on Hadoop MapReduce, a distributed big data processing framework. But, due to heavy disk I/O, MapReduce is found to be inefficient for the highly iterative FIM algorithms. Therefore, Spark, a more efficient distributed data processing framework, has been developed with in-memory computation and resilient distributed dataset (RDD) features to support the iterative algorithms. On this framework, Apriori and FP-Growth based FIM algorithms have been designed on the Spark RDD framework, but Eclat-based algorithm has not been explored yet. In this paper, RDD-Eclat, a parallel Eclat algorithm on the Spark RDD framework is proposed with its five variants. The proposed algorithms are evaluated on the various benchmark datasets, and the experimental results show that RDD-Eclat outperforms the Spark-based Apriori by many times. Also, the experimental results show the scalability of the proposed algorithms on increasing the number of cores and size of the dataset.


page 18

page 19

page 20

page 21

page 22


RDD-Eclat: Approaches to Parallelize Eclat Algorithm on Spark RDD Framework

Initially, a number of frequent itemset mining (FIM) algorithms have bee...

A Data Structure Perspective to the RDD-based Apriori Algorithm on Spark

During the recent years, a number of efficient and scalable frequent ite...

Processing Database Joins over a Shared-Nothing System of Multicore Machines

To process a large volume of data, modern data management systems use a ...

Distributed ReliefF based Feature Selection in Spark

Feature selection (FS) is a key research area in the machine learning an...

Reactive Liquid: Optimized Liquid Architecture for Elastic and Resilient Distributed Data Processing

Today's most prominent IT companies are built on the extraction of insig...

A Hierarchical Distributed Processing Framework for Big Image Data

This paper introduces an effective processing framework nominated ICP (I...

Interactive Log Parsing via Light-weight User Feedbacks

Template mining is one of the foundational tasks to support log analysis...

Please sign up or login with your details

Forgot password? Click here to reset