Differentially Private Adaptive Optimization with Delayed Preconditioners

by   Tian Li, et al.

Privacy noise may negate the benefits of using adaptive optimizers in differentially private model training. Prior works typically address this issue by using auxiliary information (e.g., public data) to boost the effectiveness of adaptive optimization. In this work, we explore techniques to estimate and efficiently adapt to gradient geometry in private adaptive optimization without auxiliary data. Motivated by the observation that adaptive methods can tolerate stale preconditioners, we propose differentially private adaptive training with delayed preconditioners (DP^2), a simple method that constructs delayed but less noisy preconditioners to better realize the benefits of adaptivity. Theoretically, we provide convergence guarantees for our method for both convex and non-convex problems, and analyze trade-offs between delay and privacy noise reduction. Empirically, we explore DP^2 across several real-world datasets, demonstrating that it can improve convergence speed by as much as 4x relative to non-adaptive baselines and match the performance of state-of-the-art optimization methods that require auxiliary data.


page 1

page 2

page 3

page 4


Private Adaptive Optimization with Side Information

Adaptive optimization methods have become the default solvers for many m...

Adaptive Private-K-Selection with Adaptive K and Application to Multi-label PATE

We provide an end-to-end Renyi DP based-framework for differentially pri...

Optimal Differentially Private Learning with Public Data

Differential Privacy (DP) ensures that training a machine learning model...

Differentially Private Learning with Per-Sample Adaptive Clipping

Privacy in AI remains a topic that draws attention from researchers and ...

Leveraging gradient-derived metrics for data selection and valuation in differentially private training

Obtaining high-quality data for collaborative training of machine learni...

DP-MAC: The Differentially Private Method of Auxiliary Coordinates for Deep Learning

Developing a differentially private deep learning algorithm is challengi...

Please sign up or login with your details

Forgot password? Click here to reset