Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime

06/20/2022
by   Leonardo Cunha, et al.
0

The recently developed average-case analysis of optimization methods allows a more fine-grained and representative convergence analysis than usual worst-case results. In exchange, this analysis requires a more precise hypothesis over the data generating process, namely assuming knowledge of the expected spectral distribution (ESD) of the random matrix associated with the problem. This work shows that the concentration of eigenvalues near the edges of the ESD determines a problem's asymptotic average complexity. This a priori information on this concentration is a more grounded assumption than complete knowledge of the ESD. This approximate concentration is effectively a middle ground between the coarseness of the worst-case scenario convergence and the restrictive previous average-case analysis. We also introduce the Generalized Chebyshev method, asymptotically optimal under a hypothesis on this concentration and globally optimal when the ESD follows a Beta distribution. We compare its performance to classical optimization algorithms, such as gradient descent or Nesterov's scheme, and we show that, in the average-case context, Nesterov's method is universally nearly optimal asymptotically.

READ FULL TEXT

page 1

page 2

page 3

page 4

03/26/2021

Automated Worst-Case Performance Analysis of Decentralized Gradient Descent

We develop a methodology to automatically compute worst-case performance...
06/08/2020

Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis

Average-case analysis computes the complexity of an algorithm averaged o...
11/03/2021

Average complexity of matrix reduction for clique filtrations

We study the algorithmic complexity of computing persistent homology of ...
01/11/2022

PEPit: computer-assisted worst-case analyses of first-order optimization methods in Python

PEPit is a Python package aiming at simplifying the access to worst-case...
05/16/2023

The Average Rate of Convergence of the Exact Line Search Gradient Descent Method

It is very well-known that when the exact line search gradient descent m...
10/19/2020

Concentration of solutions to random equations with concentration of measure hypotheses

We propose here to study the concentration of random objects that are im...
06/22/2020

Good linear classifiers are abundant in the interpolating regime

Within the machine learning community, the widely-used uniform convergen...

Please sign up or login with your details

Forgot password? Click here to reset