Characterization of Convex Objective Functions and Optimal Expected Convergence Rates for SGD

10/09/2018
by   Marten van Dijk, et al.
0

We study Stochastic Gradient Descent (SGD) with diminishing step sizes for convex objective functions. We introduce a definitional framework and theory that defines and characterizes a core property, called curvature, of convex objective functions. In terms of curvature we can derive a new inequality that can be used to compute an optimal sequence of diminishing step sizes by solving a differential equation. Our exact solutions confirm known results in literature and allows us to fully characterize a new regularizer with its corresponding expected convergence rates.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
04/02/2019

Convergence rates for the stochastic gradient descent method for non-convex objective functions

We prove the local convergence to minima and estimates on the rate of co...
research
02/03/2019

Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions

Stochastic gradient descent (SGD) is a popular and efficient method with...
research
10/10/2018

Tight Dimension Independent Lower Bound on Optimal Expected Convergence Rate for Diminishing Step Sizes in SGD

We study convergence of Stochastic Gradient Descent (SGD) for strongly c...
research
04/08/2020

Continuous and Discrete-Time Analysis of Stochastic Gradient Descent for Convex and Non-Convex Functions

This paper proposes a thorough theoretical analysis of Stochastic Gradie...
research
04/25/2017

A decentralized proximal-gradient method with network independent step-sizes and separated convergence rates

This paper considers the problem of decentralized optimization with a co...
research
06/28/2023

Ordering for Non-Replacement SGD

One approach for reducing run time and improving efficiency of machine l...
research
03/30/2023

Exact Characterization of the Convex Hulls of Reachable Sets

We study the convex hulls of reachable sets of nonlinear systems with bo...

Please sign up or login with your details

Forgot password? Click here to reset