DeepAI AI Chat
Log In Sign Up

Fast, Provably convergent IRLS Algorithm for p-norm Linear Regression

by   Deeksha Adil, et al.
Georgia Institute of Technology

Linear regression in ℓ_p-norm is a canonical optimization problem that arises in several applications, including sparse recovery, semi-supervised learning, and signal processing. Generic convex optimization algorithms for solving ℓ_p-regression are slow in practice. Iteratively Reweighted Least Squares (IRLS) is an easy to implement family of algorithms for solving these problems that has been studied for over 50 years. However, these algorithms often diverge for p > 3, and since the work of Osborne (1985), it has been an open problem whether there is an IRLS algorithm that is guaranteed to converge rapidly for p > 3. We propose p-IRLS, the first IRLS algorithm that provably converges geometrically for any p ∈ [2,∞). Our algorithm is simple to implement and is guaranteed to find a (1+ε)-approximate solution in O(p^3.5 m^p-2/2(p-1)m/ε) < O_p(√(m)m/ε ) iterations. Our experiments demonstrate that it performs even better than our theoretical bounds, beats the standard Matlab/CVX implementation for solving these problems by 10--50x, and is the fastest among available implementations in the high-accuracy regime.


page 1

page 2

page 3

page 4


Fast Algorithms for ℓ_p-Regression

The ℓ_p-norm regression problem is a classic problem in optimization wit...

Semi-Infinite Linear Regression and Its Applications

Finite linear least squares is one of the core problems of numerical lin...

Boosting as Frank-Wolfe

Some boosting algorithms, such as LPBoost, ERLPBoost, and C-ERLPBoost, a...

Improved Convergence for ℓ_∞ and ℓ_1 Regression via Iteratively Reweighted Least Squares

The iteratively reweighted least squares method (IRLS) is a popular tech...

An homotopy method for ℓ_p regression provably beyond self-concordance and in input-sparsity time

We consider the problem of linear regression where the ℓ_2^n norm loss (...

Alternating Minimization Converges Super-Linearly for Mixed Linear Regression

We address the problem of solving mixed random linear equations. We have...