Shuffled Model of Federated Learning: Privacy, Communication and Accuracy Trade-offs

by   Antonious M. Girgis, et al.

We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework. Unique challenges to the traditional ERM problem in the context of FL include (i) need to provide privacy guarantees on clients' data, (ii) compress the communication between clients and the server, since clients might have low-bandwidth links, (iii) work with a dynamic client population at each round of communication between the server and the clients, as a small fraction of clients are sampled at each round. To address these challenges we develop (optimal) communication-efficient schemes for private mean estimation for several ℓ_p spaces, enabling efficient gradient aggregation for each iteration of the optimization solution of the ERM. We also provide lower and upper bounds for mean estimation with privacy and communication constraints for arbitrary ℓ_p spaces. To get the overall communication, privacy, and optimization performance operation point, we combine this with privacy amplification opportunities inherent to this setup. Our solution takes advantage of the inherent privacy amplification provided by client sampling and data sampling at each client (through Stochastic Gradient Descent) as well as the recently developed privacy framework using anonymization, which effectively presents to the server responses that are randomly shuffled with respect to the clients. Putting these together, we demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication, but at a much lower communication cost, i.e., effectively getting communication efficiency for "free".


page 1

page 2

page 3

page 4


FedSampling: A Better Sampling Strategy for Federated Learning

Federated learning (FL) is an important technique for learning models fr...

Communication-Efficient Federated Learning through Importance Sampling

The high communication cost of sending model updates from the clients to...

Renyi Differential Privacy of the Subsampled Shuffle Model in Distributed Learning

We study privacy in a distributed learning framework, where clients coll...

Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-off in Distributed Mean Estimation

Privacy and communication constraints are two major bottlenecks in feder...

Private Federated Frequency Estimation: Adapting to the Hardness of the Instance

In federated frequency estimation (FFE), multiple clients work together ...

Exact Support Recovery in Federated Regression with One-shot Communication

Federated learning provides a framework to address the challenges of dis...

Federated Automatic Differentiation

Federated learning (FL) is a general framework for learning across heter...

Please sign up or login with your details

Forgot password? Click here to reset