Accelerated First-Order Optimization under Nonlinear Constraints

by   Michael Muehlebach, et al.

We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated first-order algorithms for constrained optimization. Unlike Frank-Wolfe or projected gradients, these algorithms avoid optimization over the entire feasible set at each iteration. We prove convergence to stationary points even in a nonconvex setting and we derive rates for the convex setting. An important property of these algorithms is that constraints are expressed in terms of velocities instead of positions, which naturally leads to sparse, local and convex approximations of the feasible set (even if the feasible set is nonconvex). Thus, the complexity tends to grow mildly in the number of decision variables and in the number of constraints, which makes the algorithms suitable for machine learning applications. We apply our algorithms to a compressed sensing and a sparse regression problem, showing that we can treat nonconvex ℓ^p constraints (p<1) efficiently, while recovering state-of-the-art performance for p=1.


On Constraints in First-Order Optimization: A View from Non-Smooth Dynamical Systems

We introduce a class of first-order methods for smooth constrained optim...

A Feasible Level Proximal Point Method for Nonconvex Sparse Constrained Optimization

Nonconvex sparse models have received significant attention in high-dime...

Structured Nonconvex and Nonsmooth Optimization: Algorithms and Iteration Complexity Analysis

Nonconvex and nonsmooth optimization problems are frequently encountered...

An efficient nonconvex reformulation of stagewise convex optimization problems

Convex optimization problems with staged structure appear in several con...

A Single-Loop Gradient Descent and Perturbed Ascent Algorithm for Nonconvex Functional Constrained Optimization

Nonconvex constrained optimization problems can be used to model a numbe...

Online Learning under Adversarial Nonlinear Constraints

In many applications, learning systems are required to process continuou...

Functional Constrained Optimization for Risk Aversion and Sparsity Control

Risk and sparsity requirements often need to be enforced simultaneously ...

Please sign up or login with your details

Forgot password? Click here to reset