A New Perspective on Boosting in Linear Regression via Subgradient Optimization and Relatives

by   Robert M. Freund, et al.

In this paper we analyze boosting algorithms in linear regression from a new perspective: that of modern first-order methods in convex optimization. We show that classic boosting algorithms in linear regression, namely the incremental forward stagewise algorithm (FS_ε) and least squares boosting (LS-Boost(ε)), can be viewed as subgradient descent to minimize the loss function defined as the maximum absolute correlation between the features and residuals. We also propose a modification of FS_ε that yields an algorithm for the Lasso, and that may be easily extended to an algorithm that computes the Lasso path for different values of the regularization parameter. Furthermore, we show that these new algorithms for the Lasso may also be interpreted as the same master algorithm (subgradient descent), applied to a regularized version of the maximum absolute correlation loss function. We derive novel, comprehensive computational guarantees for several boosting algorithms in linear regression (including LS-Boost(ε) and FS_ε) by using techniques of modern first-order methods in convex optimization. Our computational guarantees inform us about the statistical properties of boosting algorithms. In particular they provide, for the first time, a precise theoretical description of the amount of data-fidelity and regularization imparted by running a boosting algorithm with a prespecified learning rate for a fixed but arbitrary number of iterations, for any dataset.


page 1

page 2

page 3

page 4


AdaBoost and Forward Stagewise Regression are First-Order Convex Optimization Methods

Boosting methods are highly popular and effective supervised learning me...

SAGA and Restricted Strong Convexity

SAGA is a fast incremental gradient method on the finite sum problem and...

Functional Frank-Wolfe Boosting for General Loss Functions

Boosting is a generic learning method for classification and regression....

Localized Linear Regression in Networked Data

The network Lasso (nLasso) has been proposed recently as an efficient le...

Behavior of linear L2-boosting algorithms in the vanishing learning rate asymptotic

We investigate the asymptotic behaviour of gradient boosting algorithms ...

OpReg-Boost: Learning to Accelerate Online Algorithms with Operator Regression

This paper presents a new regularization approach – termed OpReg-Boost –...

MP-Boost: Minipatch Boosting via Adaptive Feature and Observation Sampling

Boosting methods are among the best general-purpose and off-the-shelf ma...

Please sign up or login with your details

Forgot password? Click here to reset