The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models

04/29/2022

∙

We study the Stochastic Gradient Descent (SGD) algorithm in nonparametric statistics: kernel regression in particular. The directional bias property of SGD, which is known in the linear regression setting, is generalized to the kernel regression. More specifically, we prove that SGD with moderate and annealing step-size converges along the direction of the eigenvector that corresponds to the largest eigenvalue of the Gram matrix. In addition, the Gradient Descent (GD) with a moderate or small step-size converges along the direction that corresponds to the smallest eigenvalue. These facts are referred to as the directional bias properties; they may interpret how an SGD-computed estimator has a potentially smaller generalization error than a GD-computed estimator. The application of our theory is demonstrated by simulation studies and a case study that is based on the FashionMNIST dataset.

READ FULL TEXT

The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

On the Double Descent of Random Features Models Trained with SGD

Cadre Modeling: Simultaneously Discovering Subpopulations and Predictive Models

On the different regimes of Stochastic Gradient Descent

The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning

Characterizing Implicit Bias in Terms of Optimization Geometry

The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models

Related Research

Direction Matters: On the Implicit Regularization Effect of Stochastic Gradient Descent with Moderate Learning Rate

Parameter Averaging for SGD Stabilizes the Implicit Bias towards Flat Regions

On the Double Descent of Random Features Models Trained with SGD

Cadre Modeling: Simultaneously Discovering Subpopulations and Predictive Models

On the different regimes of Stochastic Gradient Descent

The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning

Characterizing Implicit Bias in Terms of Optimization Geometry