Credibility evaluation of income data with hierarchical correlation reconstruction

by   Jarek Duda, et al.

In situations like tax declarations or analyzes of household budgets we would like to evaluate credibility of exogenous variable (declared income) based on some available (endogenous) variables - we want to build a model and train it on provided data sample to predict (conditional) probability distribution of exogenous variable based on values of endogenous variables. Using Polish household budget survey data there will be discussed simple and systematic adaptation of hierarchical correlation reconstruction (HCR) technique for this purpose, which allows to combine interpretability of statistics with modelling of complex densities like in machine learning. For credibility evaluation we normalize marginal distribution of predicted variable to ρ≈ 1 uniform distribution on [0,1] using empirical distribution function (x=EDF(y)∈[0,1]), then model density of its conditional distribution (Pr(x_0|x_1 x_2...)) as a linear combination of orthonormal polynomials using coefficients modelled as linear combinations of properties of the remaining variables. These coefficients can be calculated independently, have similar interpretation as cumulants, additionally allowing to directly reconstruct probability distribution. Values corresponding to high predicted density can be considered as credible, while low density suggests disagreement with statistics of data sample, for example to mark for manual verification a chosen percentage of data points evaluated as the least credible.


page 1

page 3


Modelling bid-ask spread conditional distributions using hierarchical correlation reconstruction

While we would like to predict exact values, available incomplete inform...

Exploiting statistical dependencies of time series with hierarchical correlation reconstruction

While we are usually focused on predicting future values of time series,...

Hierarchical correlation reconstruction with missing data

Machine learning often needs to estimate density from a multidimensional...

Hierarchical correlation reconstruction with missing data, for example for biology-inspired neuron

Machine learning often needs to estimate density from a multidimensional...

Low cost prediction of probability distributions of molecular properties for early virtual screening

While there is a general focus on predictions of values, mathematically ...

Sampling from Arbitrary Functions via PSD Models

In many areas of applied statistics and machine learning, generating an ...

A Variational Approach to Parameter Estimation for Characterizing 2-D Cluster Variation Method Topographies

One of the biggest challenges in characterizing 2-D topographies is succ...

Please sign up or login with your details

Forgot password? Click here to reset