Direct Runge-Kutta Discretization Achieves Acceleration
share
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient. When the function is smooth enough, we show that acceleration can be achieved by a stable discretization of the ODE using standard Runge-Kutta integrators. Specifically, we prove that under Lipschitz-gradient, convexity, and order-(s+2) differentiability assumptions, the sequence of iterates generated by discretizing the proposed second-order ODE converges to the optimal solution at a rate of O(N^-2s/s+1), where s is the order of the Runge-Kutta numerical integrator. By increasing s, the convergence rate of our method approaches the optimal rate of O(N^-2). Furthermore, we introduce a new local flatness condition on the objective, according to which rates even faster than (N^-2) can be achieved with low-order integrators and only gradient information. Notably, this flatness condition is satisfied by several standard loss functions used in machine learning, and it may be of broader independent interest. We provide numerical experiments that verify the theoretical rates predicted by our results.
READ FULL TEXT