Sharper bounds for uniformly stable algorithms

by   Olivier Bousquet, et al.

The generalization bounds for stable algorithms is a classical question in learning theory taking its roots in the early works of Vapnik and Chervonenkis and Rogers and Wagner. In a series of recent breakthrough papers, Feldman and Vondrak have shown that the best known high probability upper bounds for uniformly stable learning algorithms due to Bousquet and Elisseeff are sub-optimal in some natural regimes. To do so, they proved two generalization bounds that significantly outperform the original generalization bound. Feldman and Vondrak also asked if it is possible to provide sharper bounds and prove corresponding high probability lower bounds. This paper is devoted to these questions: firstly, inspired by the original arguments of, we provide a short proof of the moment bound that implies the generalization bound stronger than both recent results. Secondly, we prove general lower bounds, showing that our moment bound is sharp (up to a logarithmic factor) unless some additional properties of the corresponding random variables are used. Our main probabilistic result is a general concentration inequality for weakly correlated random variables, which may be of independent interest.


page 1

page 2

page 3

page 4


Generalization Bounds for Uniformly Stable Algorithms

Uniform stability of a learning algorithm is a classical notion of algor...

New Bounds for the Vertices of the Integer Hull

The vertices of the integer hull are the integral equivalent to the well...

Singletons for Simpletons: Revisiting Windowed Backoff using Chernoff Bounds

For the well-known problem of balls dropped uniformly at random into bin...

High probability generalization bounds for uniformly stable algorithms with nearly optimal rate

Algorithmic stability is a classical approach to understanding and analy...

Scalar and Matrix Chernoff Bounds from ℓ_∞-Independence

We present new scalar and matrix Chernoff-style concentration bounds for...

Solvable Integration Problems and Optimal Sample Size Selection

We want to compute the integral of a function or the expectation of a ra...

Lower Bound for RIP Constants and Concentration of Sum of Top Order Statistics

Restricted Isometry Property (RIP) is of fundamental importance in the t...

Please sign up or login with your details

Forgot password? Click here to reset