FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizer by Strong Convexity

04/28/2021

∙

The AdaBelief algorithm demonstrates superior generalization ability to the Adam algorithm by viewing the exponential moving average of observed gradients. AdaBelief is proved to have a data-dependent O(√(T)) regret bound when objective functions are convex, where T is a time horizon. However, it remains to be an open problem on how to exploit strong convexity to further improve the convergence rate of AdaBelief. To tackle this problem, we present a novel optimization algorithm under strong convexity, called FastAdaBelief. We prove that FastAdaBelief attains a data-dependant O(log T) regret bound, which is substantially lower than AdaBelief. In addition, the theoretical analysis is validated by extensive experiments performed on open datasets (i.e., CIFAR-10 and Penn Treebank) for image classification and language modeling.

READ FULL TEXT

FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizer by Strong Convexity

SAdam: A Variant of Adam for Strongly Convex Functions

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate

Stochastic Composite Least-Squares Regression with convergence rate O(1/n)

Accelerated Convergence of Nesterov's Momentum for Deep Neural Networks under Partial Strong Convexity

Adam revisited: a weighted past gradients perspective

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

On the Global Linear Convergence of Frank-Wolfe Optimization Variants

FastAdaBelief: Improving Convergence Rate for Belief-based Adaptive Optimizer by Strong Convexity

Related Research

SAdam: A Variant of Adam for Strongly Convex Functions

Stochastic Approximation of Smooth and Strongly Convex Functions: Beyond the O(1/T) Convergence Rate

Stochastic Composite Least-Squares Regression with convergence rate O(1/n)

Accelerated Convergence of Nesterov's Momentum for Deep Neural Networks under Partial Strong Convexity

Adam revisited: a weighted past gradients perspective

A Two-Time-Scale Stochastic Optimization Framework with Applications in Control and Reinforcement Learning

On the Global Linear Convergence of Frank-Wolfe Optimization Variants