Improving Distributed Similarity Join in Metric Space with Error-bounded Sampling

by   Jiacheng Wu, et al.

Given two sets of objects, metric similarity join finds all similar pairs of objects according to a particular distance function in metric space. There is an increasing demand to provide a scalable similarity join framework which can support efficient query and analytical services in the era of Big Data. The existing distributed metric similarity join algorithms adopt random sampling techniques to produce pivots and utilize holistic partitioning methods based on the generated pivots to partition data, which results in data skew problem since both the generated pivots and the partition strategies have no quality guarantees. To address the limitation, we propose SP-Join, an end-to-end framework to support distributed similarity join in metric space based on the MapReduce paradigm, which (i) employs an estimation-based stratified sampling method to produce pivots with quality guarantees for any sample size, and (ii) devises an effective cost model as the guideline to split the whole datasets into partition in map and reduce phases according to the sampled pivots. Although obtaining an optimal set of partitions is proven to be NP-Hard, SP-Join adopts efficient partitioning strategies based on such a cost model to achieve an even partition with explainable quality. We implement our framework upon Apache Spark platform and conduct extensive experiments on four real-world datasets. The results show that our method significantly outperforms state-of-the-art methods.


page 1

page 2

page 3

page 4


Error-Tolerant Big Data Processing

Real-world data contains various kinds of errors. Before analyzing data,...

Similarity Join and Similarity Self-Join Size Estimation in a Streaming Environment

We study the problem of similarity self-join and similarity join size es...

Cost-efficient Data Acquisition on Online Data Marketplaces for Correlation Analysis

Incentivized by the enormous economic profits, the data marketplace plat...

Towards Fast Theta-join: A Prefiltering and Amalgamated Partitioning Approach

As one of the most useful online processing techniques, the theta-join o...

Near-Optimal Distributed Band-Joins through Recursive Partitioning

We consider running-time optimization for band-joins in a distributed sy...

Measuring and Predicting the Quality of a Join for Data Discovery

We study the problem of discovering joinable datasets at scale. We appro...

Similarity Join and Self-Join Size Estimation in a Streaming Environment

We study the problem of similarity self-join and similarity join size es...

Please sign up or login with your details

Forgot password? Click here to reset