DeepAI AI Chat
Log In Sign Up

Scalable Derivative-Free Optimization for Nonlinear Least-Squares Problems

by   Coralia Cartia, et al.

Derivative-free - or zeroth-order - optimization (DFO) has gained recent attention for its ability to solve problems in a variety of application areas, including machine learning, particularly involving objectives which are stochastic and/or expensive to compute. In this work, we develop a novel model-based DFO method for solving nonlinear least-squares problems. We improve on state-of-the-art DFO by performing dimensionality reduction in the observational space using sketching methods, avoiding the construction of a full local model. Our approach has a per-iteration computational cost which is linear in problem dimension in a big data regime, and numerical evidence demonstrates that, compared to existing software, it has dramatically improved runtime performance on overdetermined least-squares problems.


page 1

page 2

page 3

page 4


Accelerated derivative-free nonlinear least-squares applied to the estimation of Manning coefficients

A general framework for solving nonlinear least squares problems without...

Covariance-free Partial Least Squares: An Incremental Dimensionality Reduction Method

Dimensionality reduction plays an important role in computer vision prob...

Nonlinear Least Squares for Large-Scale Machine Learning using Stochastic Jacobian Estimates

For large nonlinear least squares loss functions in machine learning we ...

ZOOpt/ZOOjl: Toolbox for Derivative-Free Optimization

Recent advances of derivative-free optimization allow efficient approxim...

A Randomised Subspace Gauss-Newton Method for Nonlinear Least-Squares

We propose a Randomised Subspace Gauss-Newton (R-SGN) algorithm for solv...

Adaptive Least-Squares Methods for Convection-Dominated Diffusion-Reaction Problems

This paper studies adaptive least-squares finite element methods for con...

Derivative-Free Multiobjective Trust Region Descent Method Using Radial Basis Function Surrogate Models

We present a flexible trust region descend algorithm for unconstrained a...