Goodness-of-Fit Tests for Large Datasets

by   Taras Lazariv, et al.

Nowadays, data analysis in the world of Big Data is connected typically to data mining, descriptive or exploratory statistics, e. g. cluster analysis, classification or regression analysis. Aside these techniques there is a huge area of methods from inferential statistics that are rarely considered in connection with Big Data. Nevertheless, inferential methods are also of use for Big Data analysis, especially for quantifying uncertainty. The article at hand will provide some insights to methodological and technical issues referring inferential methods in the Big Data area in order to bring together Big Data and inferential statistics, as it comes along with its difficulties. We present an approach that allows testing goodness-of-fit without model assumptions and relying on the empirical distribution. Especially, the method is able to utilize information from large datasets. Thereby, the approach is based on a clear theoretical background. We concentrate on the widely-used Kolmogorov-Smirnov test that is applied for testing goodness-of-fit in statistics. Our approach can be parallelized easily, which makes it applicable to distributed datasets particularly on a compute cluster. By this contribution, we turn to an audience that is interested in the technical and methodological backgrounds while implementing especially inferential statistical methods with Big Data tools as e. g. Spark.


Challenges of Big Data Analysis

Big Data bring new opportunities to modern society and challenges to dat...

Analyzing Big Datasets of Genomic Sequences: Fast and Scalable Collection of k-mer Statistics

Distributed approaches based on the map-reduce programming paradigm have...

Simulating Name-like Vectors for Testing Large-scale Entity Resolution

Accurate and efficient entity resolution (ER) has been a problem in data...

The Role of Data Analysis in the Development of Intelligent Energy Networks

Data analysis plays an important role in the development of intelligent ...

What is the best predictor that you can compute in five minutes using a given Bayesian hierarchical model?

The goal of this paper is to provide a way for statisticians to answer t...

Gap-Measure Tests with Applications to Data Integrity Verification

In this paper we propose and examine gap statistics for assessing unifor...

Please sign up or login with your details

Forgot password? Click here to reset