On the equivalence of different adaptive batch size selection strategies for stochastic gradient descent methods

by   Luis Espath, et al.

In this study, we demonstrate that the norm test and inner product/orthogonality test presented in <cit.> are equivalent in terms of the convergence rates associated with Stochastic Gradient Descent (SGD) methods if ϵ^2=θ^2+ν^2 with specific choices of θ and ν. Here, ϵ controls the relative statistical error of the norm of the gradient while θ and ν control the relative statistical error of the gradient in the direction of the gradient and in the direction orthogonal to the gradient, respectively. Furthermore, we demonstrate that the inner product/orthogonality test can be as inexpensive as the norm test in the best case scenario if θ and ν are optimally selected, but the inner product/orthogonality test will never be more computationally affordable than the norm test if ϵ^2=θ^2+ν^2. Finally, we present two stochastic optimization problems to illustrate our results.


page 1

page 2

page 3

page 4


SGD Implicitly Regularizes Generalization Error

We derive a simple and model-independent formula for the change in the g...

Adaptive Sampling Quasi-Newton Methods for Zeroth-Order Stochastic Optimization

We consider unconstrained stochastic optimization problems with no avail...

Randomized Stochastic Gradient Descent Ascent

An increasing number of machine learning problems, such as robust or adv...

Adaptive Sampling Strategies for Stochastic Optimization

In this paper, we propose a stochastic optimization method that adaptive...

On Convergence-Diagnostic based Step Sizes for Stochastic Gradient Descent

Constant step-size Stochastic Gradient Descent exhibits two phases: a tr...

Low-Complexity Iterative Methods for Complex-Variable Matrix Optimization Problems in Frobenius Norm

Complex-variable matrix optimization problems (CMOPs) in Frobenius norm ...

Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum

Convergence detection of iterative stochastic optimization methods is of...

Please sign up or login with your details

Forgot password? Click here to reset