PNKH-B: A Projected Newton-Krylov Method for Large-Scale Bound-Constrained Optimization

by   Kelvin Kan, et al.

We present PNKH-B, a projected Newton-Krylov method with a low-rank approximated Hessian metric for approximately solving large-scale optimization problems with bound constraints. PNKH-B is geared toward situations in which function and gradient evaluations are expensive, and the (approximate) Hessian is only available through matrix-vector products. This is commonly the case in large-scale parameter estimation, machine learning, and image processing. In each iteration, PNKH-B generates a low-rank approximation of the (approximate) Hessian using Lanczos tridiagonalization and then solves a quadratic projection problem to update the iterate. The key idea is to compute the projection with respect to the norm defined by the low-rank approximation. Hence, PNKH-B can be viewed as a projected variable metric method. We present an interior point method to solve the quadratic projection problem efficiently. Since the interior point method effectively exploits the low-rank structure, its computational cost only scales linearly with respect to the number of variables, and it only adds negligible computational time. We also experiment with variants of PNKH-B that incorporate estimates of the active set into the Hessian approximation. We prove the global convergence to a stationary point under standard assumptions. Using three numerical experiments motivated by parameter estimation, machine learning, and image reconstruction, we show that the consistent use of the Hessian metric in PNKH-B leads to fast convergence, particularly in the first few iterations. We provide our MATLAB implementation at


page 15

page 18


Structured Stochastic Quasi-Newton Methods for Large-Scale Optimization Problems

In this paper, we consider large-scale finite-sum nonconvex problems ari...

A Riemannian Newton Optimization Framework for the Symmetric Tensor Rank Approximation Problem

The symmetric tensor rank approximation problem (STA) consists in comput...

Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates

For large nonlinear least squares loss functions in machine learning we ...

A Random-Feature Based Newton Method for Empirical Risk Minimization in Reproducing Kernel Hilbert Space

In supervised learning using kernel methods, we encounter a large-scale ...

Low Rank Saddle Free Newton: Algorithm and Analysis

Many tasks in engineering fields and machine learning involve minimizing...

Fast and stable modification of the Gauss-Newton method for low-rank signal estimation

The weighted nonlinear least-squares problem for low-rank signal estimat...

Efficient Second-Order Plane Adjustment

Planes are generally used in 3D reconstruction for depth sensors, such a...

Please sign up or login with your details

Forgot password? Click here to reset