Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

by   Varun Kanade, et al.

The local Rademacher complexity framework is one of the most successful general-purpose toolboxes for establishing sharp excess risk bounds for statistical estimators based on the framework of empirical risk minimization. Applying this toolbox typically requires using the Bernstein condition, which often restricts applicability to convex and proper settings. Recent years have witnessed several examples of problems where optimal statistical performance is only achievable via non-convex and improper estimators originating from aggregation theory, including the fundamental problem of model selection. These examples are currently outside of the reach of the classical localization theory. In this work, we build upon the recent approach to localization via offset Rademacher complexities, for which a general high-probability theory has yet to be established. Our main result is an exponential-tail excess risk bound expressed in terms of the offset Rademacher complexity that yields results at least as sharp as those obtainable via the classical theory. However, our bound applies under an estimator-dependent geometric condition (the "offset condition") instead of the estimator-independent (but, in general, distribution-dependent) Bernstein condition on which the classical theory relies. Our results apply to improper prediction regimes not directly covered by the classical theory.


page 1

page 2

page 3

page 4


Localization, Convexity, and Star Aggregation

Offset Rademacher complexities have been shown to imply sharp, data-depe...

Local Risk Bounds for Statistical Aggregation

In the problem of aggregation, the aim is to combine a given class of ba...

Non-Asymptotic Analysis of Excess Risk via Empirical Risk Landscape

In this paper, we provide a unified analysis of the excess risk of the m...

Learning with Square Loss: Localization through Offset Rademacher Complexity

We consider regression with square loss and general classes of functions...

PAC-Bayesian Bounds for Randomized Empirical Risk Minimizers

The aim of this paper is to generalize the PAC-Bayesian theorems proved ...

Margin-adaptive model selection in statistical learning

A classical condition for fast learning rates is the margin condition, f...

Instance-dependent uniform tail bounds for empirical processes

We formulate a uniform tail bound for empirical processes indexed by a c...

Please sign up or login with your details

Forgot password? Click here to reset