Instance-Dependent Generalization Bounds via Optimal Transport

by   Songyan Hou, et al.

Existing generalization bounds fail to explain crucial factors that drive generalization of modern neural networks. Since such bounds often hold uniformly over all parameters, they suffer from over-parametrization, and fail to account for the strong inductive bias of initialization and stochastic gradient descent. As an alternative, we propose a novel optimal transport interpretation of the generalization problem. This allows us to derive instance-dependent generalization bounds that depend on the local Lipschitz regularity of the earned prediction function in the data space. Therefore, our bounds are agnostic to the parametrization of the model and work well when the number of training samples is much smaller than the number of parameters. With small modifications, our approach yields accelerated rates for data on low-dimensional manifolds, and guarantees under distribution shifts. We empirically analyze our generalization bounds for neural networks, showing that the bound values are meaningful and capture the effect of popular regularization methods during training.


Measuring Generalization with Optimal Transport

Understanding the generalization of deep neural networks is one of the m...

A Principle of Least Action for the Training of Neural Networks

Neural networks have been achieving high generalization performance on m...

GAN Estimation of Lipschitz Optimal Transport Maps

This paper introduces the first statistically consistent estimator of th...

How You Start Matters for Generalization

Characterizing the remarkable generalization properties of over-paramete...

Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks

Motivated by the learned iterative soft thresholding algorithm (LISTA), ...

Improving Generative Flow Networks with Path Regularization

Generative Flow Networks (GFlowNets) are recently proposed models for le...

Please sign up or login with your details

Forgot password? Click here to reset