On uniform-in-time diffusion approximation for stochastic gradient descent

07/11/2022
by   Lei Li, et al.
0

The diffusion approximation of stochastic gradient descent (SGD) in current literature is only valid on a finite time interval. In this paper, we establish the uniform-in-time diffusion approximation of SGD, by only assuming that the expected loss is strongly convex and some other mild conditions, without assuming the convexity of each random loss function. The main technique is to establish the exponential decay rates of the derivatives of the solution to the backward Kolmogorov equation. The uniform-in-time approximation allows us to study asymptotic behaviors of SGD via the continuous stochastic differential equation (SDE) even when the random objective function f(·;ξ) is not strongly convex.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/02/2019

Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation

Diffusion approximation provides weak approximation for stochastic gradi...
research
10/27/2019

A geometric interpretation of stochastic gradient descent using diffusion metrics

Stochastic gradient descent (SGD) is a key ingredient in the training of...
research
07/27/2020

Stochastic Gradient Descent applied to Least Squares regularizes in Sobolev spaces

We study the behavior of stochastic gradient descent applied to Ax -b _2...
research
08/28/2020

Predicting Training Time Without Training

We tackle the problem of predicting the number of optimization steps tha...
research
04/08/2020

Continuous and Discrete-Time Analysis of Stochastic Gradient Descent for Convex and Non-Convex Functions

This paper proposes a thorough theoretical analysis of Stochastic Gradie...
research
10/13/2022

From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) has been the method of choice for lear...
research
08/25/2023

Network Embedding Using Sparse Approximations of Random Walks

In this paper, we propose an efficient numerical implementation of Netwo...

Please sign up or login with your details

Forgot password? Click here to reset