The One-Inclusion Graph Algorithm is not Always Optimal

by   Ishaq Aden-Ali, et al.
berkeley college

The one-inclusion graph algorithm of Haussler, Littlestone, and Warmuth achieves an optimal in-expectation risk bound in the standard PAC classification setup. In one of the first COLT open problems, Warmuth conjectured that this prediction strategy always implies an optimal high probability bound on the risk, and hence is also an optimal PAC algorithm. We refute this conjecture in the strongest sense: for any practically interesting Vapnik-Chervonenkis class, we provide an in-expectation optimal one-inclusion graph algorithm whose high probability risk bound cannot go beyond that implied by Markov's inequality. Our construction of these poorly performing one-inclusion graph algorithms uses Varshamov-Tenengolts error correcting codes. Our negative result has several implications. First, it shows that the same poor high-probability performance is inherited by several recent prediction strategies based on generalizations of the one-inclusion graph algorithm. Second, our analysis shows yet another statistical problem that enjoys an estimator that is provably optimal in expectation via a leave-one-out argument, but fails in the high-probability regime. This discrepancy occurs despite the boundedness of the binary loss for which arguments based on concentration inequalities often provide sharp high probability risk bounds.


page 1

page 2

page 3

page 4


Optimal PAC Bounds Without Uniform Convergence

In statistical learning theory, determining the sample complexity of rea...

PAC-Bayesian Inequalities for Martingales

We present a set of high-probability inequalities that control the conce...

High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Algorithmic stability is a classical approach to understanding and analy...

Exponential Stochastic Inequality

We develop the concept of exponential stochastic inequality (ESI), a nov...

PAC-Bayesian Bound for the Conditional Value at Risk

Conditional Value at Risk (CVaR) is a family of "coherent risk measures"...

High Probability Bounds for Stochastic Continuous Submodular Maximization

We consider maximization of stochastic monotone continuous submodular fu...

Analysis of Kelner and Levin graph sparsification algorithm for a streaming setting

We derive a new proof to show that the incremental resparsification algo...

Please sign up or login with your details

Forgot password? Click here to reset