A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity

by   Aixiang Chen, et al.

SGD (Stochastic Gradient Descent) is a popular algorithm for large scale optimization problems due to its low iterative cost. However, SGD can not achieve linear convergence rate as FGD (Full Gradient Descent) because of the inherent gradient variance. To attack the problem, mini-batch SGD was proposed to get a trade-off in terms of convergence rate and iteration cost. In this paper, a general CVI (Convergence-Variance Inequality) equation is presented to state formally the interaction of convergence rate and gradient variance. Then a novel algorithm named SSAG (Stochastic Stratified Average Gradient) is introduced to reduce gradient variance based on two techniques, stratified sampling and averaging over iterations that is a key idea in SAG (Stochastic Average Gradient). Furthermore, SSAG can achieve linear convergence rate of O((1-μ/8CL)^k) at smaller storage and iterative costs, where C≥ 2 is the category number of training data. This convergence rate depends mainly on the variance between classes, but not on the variance within the classes. In the case of C≪ N (N is the training data size), SSAG's convergence rate is much better than SAG's convergence rate of O((1-μ/8NL)^k). Our experimental results show SSAG outperforms SAG and many other algorithms.


page 1

page 2

page 3

page 4


Trading-off variance and complexity in stochastic gradient descent

Stochastic gradient descent is the method of choice for large-scale mach...

MSTGD:A Memory Stochastic sTratified Gradient Descent Method with an Exponential Convergence Rate

The fluctuation effect of gradient expectation and variance caused by pa...

Accelerating Minibatch Stochastic Gradient Descent using Typicality Sampling

Machine learning, especially deep neural networks, has been rapidly deve...

ROOT-SGD: Sharp Nonasymptotics and Asymptotic Efficiency in a Single Algorithm

The theory and practice of stochastic optimization has focused on stocha...

Stop Wasting My Gradients: Practical SVRG

We present and analyze several strategies for improving the performance ...

Solving A System Of Linear Equations By Randomized Orthogonal Projections

Randomization has shown catalyzing effects in linear algebra with promis...

Accelerated Convergence for Counterfactual Learning to Rank

Counterfactual Learning to Rank (LTR) algorithms learn a ranking model f...

Please sign up or login with your details

Forgot password? Click here to reset