Recent Theoretical Advances in Non-Convex Optimization

by   Marina Danilova, et al.

Motivated by recent increased interest in optimization algorithms for non-convex optimization in application to training deep neural networks and other optimization problems in data analysis, we give an overview of recent theoretical results on global performance guarantees of optimization algorithms for non-convex optimization. We start with classical arguments showing that general non-convex problems could not be solved efficiently in a reasonable time. Then we give a list of problems that can be solved efficiently to find the global minimizer by exploiting the structure of the problem as much as it is possible. Another way to deal with non-convexity is to relax the goal from finding the global minimum to finding a stationary point or a local minimum. For this setting, we first present known results for the convergence rates of deterministic first-order methods, which are then followed by a general theoretical analysis of optimal stochastic and randomized gradient schemes, and an overview of the stochastic first-order methods. After that, we discuss quite general classes of non-convex problems, such as minimization of α-weakly-quasi-convex functions and functions that satisfy Polyak–Lojasiewicz condition, which still allow obtaining theoretical convergence guarantees of first-order methods. Then we consider higher-order and zeroth-order/derivative-free methods and their convergence rates for non-convex optimization problems.


page 1

page 2

page 3

page 4


On The Convergence of First Order Methods for Quasar-Convex Optimization

In recent years, the success of deep learning has inspired many research...

Dimension-free convergence rates for gradient Langevin dynamics in RKHS

Gradient Langevin dynamics (GLD) and stochastic GLD (SGLD) have attracte...

Non-Convex Optimization with Certificates and Fast Rates Through Kernel Sums of Squares

We consider potentially non-convex optimization problems, for which opti...

An Efficient Framework for Global Non-Convex Polynomial Optimization over the Hypercube

We present a novel efficient theoretical and numerical framework for sol...

Weak Supermodularity Assists Submodularity-based Approaches to Non-convex Constrained Optimization

Non-convex constrained optimization problems have many applications in m...

Global Optimality in Tensor Factorization, Deep Learning, and Beyond

Techniques involving factorization are found in a wide range of applicat...

Derivative-Free Methods for Policy Optimization: Guarantees for Linear Quadratic Systems

We study derivative-free methods for policy optimization over the class ...

Please sign up or login with your details

Forgot password? Click here to reset