The Optimal Approximation Factor in Density Estimation

by   Olivier Bousquet, et al.
University of California, San Diego

Consider the following problem: given two arbitrary densities q_1,q_2 and a sample-access to an unknown target density p, find which of the q_i's is closer to p in total variation. A remarkable result due to Yatracos shows that this problem is tractable in the following sense: there exists an algorithm that uses O(ϵ^-2) samples from p and outputs q_i such that with high probability, TV(q_i,p) ≤ 3·opt + ϵ, where opt= {TV(q_1,p),TV(q_2,p)}. Moreover, this result extends to any finite class of densities Q: there exists an algorithm that outputs the best density in Q up to a multiplicative approximation factor of 3. We complement and extend this result by showing that: (i) the factor 3 can not be improved if one restricts the algorithm to output a density from Q, and (ii) if one allows the algorithm to output arbitrary densities (e.g. a mixture of densities from Q), then the approximation factor can be reduced to 2, which is optimal. In particular this demonstrates an advantage of improper learning over proper in this setup. We develop two approaches to achieve the optimal approximation factor of 2: an adaptive one and a static one. Both approaches are based on a geometric point of view of the problem and rely on estimating surrogate metrics to the total variation. Our sample complexity bounds exploit techniques from Adaptive Data Analysis.


page 1

page 2

page 3

page 4


On Approximating Total Variation Distance

Total variation distance (TV distance) is a fundamental notion of distan...

Statistical Learnability of Generalized Additive Models based on Total Variation Regularization

A generalized additive model (GAM, Hastie and Tibshirani (1987)) is a no...

Statistically Near-Optimal Hypothesis Selection

Hypothesis Selection is a fundamental distribution learning problem wher...

Efficient Density Estimation via Piecewise Polynomial Approximation

We give a highly efficient "semi-agnostic" algorithm for learning univar...

When does the Tukey median work?

We analyze the performance of the Tukey median estimator under total var...

Sequential algorithms for testing identity and closeness of distributions

What advantage do sequential procedures provide over batch algorithms fo...

An Optimal Reduction of TV-Denoising to Adaptive Online Learning

We consider the problem of estimating a function from n noisy samples wh...

Please sign up or login with your details

Forgot password? Click here to reset