A label efficient two-sample test

by   Weizhi Li, et al.

Two-sample tests evaluate whether two samples are realizations of the same distribution (the null hypothesis) or two different distributions (the alternative hypothesis). In the traditional formulation of this problem, the statistician has access to both the measurements (feature variables) and the group variable (label variable). However, in several important applications, feature variables can be easily measured but the binary label variable is unknown and costly to obtain. In this paper, we consider this important variation on the classical two-sample test problem and pose it as a problem of obtaining the labels of only a small number of samples in service of performing a two-sample test. We devise a label efficient three-stage framework: firstly, a classifier is trained with samples uniformly labeled to model the posterior probabilities of the labels; secondly, a novel query scheme dubbed bimodal query is used to query labels of samples from both classes with maximum posterior probabilities, and lastly, the classical Friedman-Rafsky (FR) two-sample test is performed on the queried samples. Our theoretical analysis shows that bimodal query is optimal for two-sample testing using the FR statistic under reasonable conditions and that the three-stage framework controls the Type I error. Extensive experiments performed on synthetic, benchmark, and application-specific datasets demonstrate that the three-stage framework has decreased Type II error over uniform querying and certainty-based querying with same number of labels while controlling the Type I error. Source code for our algorithms and experimental results is available at https://github.com/wayne0908/Label-Efficient-Two-Sample.


Active Sequential Two-Sample Testing

Two-sample testing tests whether the distributions generating two sample...

Trustable Co-label Learning from Multiple Noisy Annotators

Supervised deep learning depends on massive accurately annotated example...

CTRL: Clustering Training Losses for Label Error Detection

In supervised machine learning, use of correct labels is extremely impor...

A statistical Testing Procedure for Validating Class Labels

Motivated by an open problem of validating protein identities in label-f...

URL: Combating Label Noise for Lung Nodule Malignancy Grading

Due to the complexity of annotation and inter-annotator variability, mos...

batchboost: regularization for stabilizing training with resistance to underfitting overfitting

Overfitting underfitting and stable training are an important challe...

A computational approach to the Kiefer-Weiss problem for sampling from a Bernoulli population

We present a computational approach to solution of the Kiefer-Weiss prob...

Please sign up or login with your details

Forgot password? Click here to reset