Accelerating Inexact HyperGradient Descent for Bilevel Optimization

by   Haikuo Yang, et al.

We present a method for solving general nonconvex-strongly-convex bilevel optimization problems. Our method – the Restarted Accelerated HyperGradient Descent () method – finds an ϵ-first-order stationary point of the objective with 𝒪̃(κ^3.25ϵ^-1.75) oracle complexity, where κ is the condition number of the lower-level objective and ϵ is the desired accuracy. We also propose a perturbed variant of for finding an (ϵ,𝒪(κ^2.5√(ϵ) ))-second-order stationary point within the same order of oracle complexity. Our results achieve the best-known theoretical guarantees for finding stationary points in bilevel optimization and also improve upon the existing upper complexity bound for finding second-order stationary points in nonconvex-strongly-concave minimax optimization problems, setting a new state-of-the-art benchmark. Empirical studies are conducted to validate the theoretical results in this paper.


page 1

page 2

page 3

page 4


Finding Second-Order Stationary Point for Nonconvex-Strongly-Concave Minimax Problem

We study the smooth minimax optimization problem of the form min_ xmax_ ...

Efficiently Escaping Saddle Points in Bilevel Optimization

Bilevel optimization is one of the fundamental problems in machine learn...

Sharp oracle inequalities for stationary points of nonconvex penalized M-estimators

Many statistical estimation procedures lead to nonconvex optimization pr...

On Bilevel Optimization without Lower-level Strong Convexity

Theoretical properties of bilevel problems are well studied when the low...

Near-Optimal Fully First-Order Algorithms for Finding Stationary Points in Bilevel Optimization

Bilevel optimization has various applications such as hyper-parameter op...

SONIA: A Symmetric Blockwise Truncated Optimization Algorithm

This work presents a new algorithm for empirical risk minimization. The ...

Accelerated Zeroth-Order Momentum Methods from Mini to Minimax Optimization

In the paper, we propose a new accelerated zeroth-order momentum (Acc-ZO...

Please sign up or login with your details

Forgot password? Click here to reset