Accelerated SGD for Non-Strongly-Convex Least Squares

03/03/2022
by   Aditya Varre, et al.
0

We consider stochastic approximation for the least squares regression problem in the non-strongly convex setting. We present the first practical algorithm that achieves the optimal prediction error rates in terms of dependence on the noise of the problem, as O(d/t) while accelerating the forgetting of the initial conditions to O(d/t^2). Our new algorithm is based on a simple modification of the accelerated gradient descent. We provide convergence results for both the averaged and the last iterate of the algorithm. In order to describe the tightness of these new bounds, we present a matching lower bound in the noiseless setting and thus show the optimality of our algorithm.

READ FULL TEXT

page 1

page 2

page 3

page 4

02/17/2016

Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression

We consider the optimization of a quadratic objective function whose gra...
02/13/2020

An Optimal Multistage Stochastic Gradient Method for Minimax Problems

In this paper, we study the minimax optimization problem in the smooth a...
03/21/2021

ANITA: An Optimal Loopless Accelerated Variance-Reduced Gradient Method

We propose a novel accelerated variance-reduced gradient method called A...
01/23/2019

A Universally Optimal Multistage Accelerated Stochastic Gradient Method

We study the problem of minimizing a strongly convex and smooth function...
09/14/2023

Acceleration by Stepsize Hedging I: Multi-Step Descent and the Silver Stepsize Schedule

Can we accelerate convergence of gradient descent without changing the a...
07/04/2023

Accelerated stochastic approximation with state-dependent noise

We consider a class of stochastic smooth convex optimization problems un...
02/19/2021

Permutation-Based SGD: Is Random Optimal?

A recent line of ground-breaking results for permutation-based SGD has c...

Please sign up or login with your details

Forgot password? Click here to reset