Generalization Bounds for Stochastic Gradient Langevin Dynamics: A Unified View via Information Leakage Analysis

by   Bingzhe Wu, et al.

Recently, generalization bounds of the non-convex empirical risk minimization paradigm using Stochastic Gradient Langevin Dynamics (SGLD) have been extensively studied. Several theoretical frameworks have been presented to study this problem from different perspectives, such as information theory and stability. In this paper, we present a unified view from privacy leakage analysis to investigate the generalization bounds of SGLD, along with a theoretical framework for re-deriving previous results in a succinct manner. Aside from theoretical findings, we conduct various numerical studies to empirically assess the information leakage issue of SGLD. Additionally, our theoretical and empirical results provide explanations for prior works that study the membership privacy of SGLD.


page 1

page 2

page 3

page 4


Characterizing Membership Privacy in Stochastic Gradient Langevin Dynamics

Bayesian deep learning is recently regarded as an intrinsic way to chara...

Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints

Algorithm-dependent generalization error bounds are central to statistic...

Stability and Generalization for Markov Chain Stochastic Gradient Methods

Recently there is a large amount of work devoted to the study of Markov ...

Uniform-in-Time Wasserstein Stability Bounds for (Noisy) Stochastic Gradient Descent

Algorithmic stability is an important notion that has proven powerful fo...

Bounding Information Leakage in Machine Learning

Machine Learning services are being deployed in a large range of applica...

Differential Privacy for Pairwise Learning: Non-convex Analysis

Pairwise learning focuses on learning tasks with pairwise loss functions...

Impact of Prior Knowledge and Data Correlation on Privacy Leakage: A Unified Analysis

It has been widely understood that differential privacy (DP) can guarant...

Please sign up or login with your details

Forgot password? Click here to reset