Compressed and distributed least-squares regression: convergence rates with applications to Federated Learning

by   Constantin Philippenko, et al.

In this paper, we investigate the impact of compression on stochastic gradient algorithms for machine learning, a technique widely used in distributed and federated learning. We underline differences in terms of convergence rates between several unbiased compression operators, that all satisfy the same condition on their variance, thus going beyond the classical worst-case analysis. To do so, we focus on the case of least-squares regression (LSR) and analyze a general stochastic approximation algorithm for minimizing quadratic functions relying on a random field. We consider weak assumptions on the random field, tailored to the analysis (specifically, expected Hölder regularity), and on the noise covariance, enabling the analysis of various randomizing mechanisms, including compression. We then extend our results to the case of federated learning. More formally, we highlight the impact on the convergence of the covariance ℭ_ania of the additive noise induced by the algorithm. We demonstrate despite the non-regularity of the stochastic field, that the limit variance term scales with Tr(ℭ_ania H^-1)/K (where H is the Hessian of the optimization problem and K the number of iterations) generalizing the rate for the vanilla LSR case where it is σ^2 Tr(H H^-1) / K = σ^2 d / K (Bach and Moulines, 2013). Then, we analyze the dependency of ℭ_ania on the compression strategy and ultimately its impact on convergence, first in the centralized case, then in two heterogeneous FL frameworks.


page 1

page 2

page 3

page 4


Artemis: tight convergence guarantees for bidirectional compression in Federated Learning

We introduce a new algorithm - Artemis - tackling the problem of learnin...

On the Convergence of Local Descent Methods in Federated Learning

In federated distributed learning, the goal is to optimize a global trai...

Federated Learning with Compression: Unified Analysis and Sharp Guarantees

In federated learning, communication cost is often a critical bottleneck...

z-SignFedAvg: A Unified Stochastic Sign-based Compression for Federated Learning

Federated Learning (FL) is a promising privacy-preserving distributed le...

Learning Linear Dynamical Systems with Semi-Parametric Least Squares

We analyze a simple prefiltered variation of the least squares estimator...

Federated Learning Using Variance Reduced Stochastic Gradient for Probabilistically Activated Agents

This paper proposes an algorithm for Federated Learning (FL) with a two-...

Please sign up or login with your details

Forgot password? Click here to reset