Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning
share
We study the asynchronous stochastic gradient descent algorithm for distributed training over n workers which have varying computation and communication frequency over time. In this algorithm, workers compute stochastic gradients in parallel at their own pace and return those to the server without any synchronization. Existing convergence rates of this algorithm for non-convex smooth objectives depend on the maximum gradient delay τ_max and show that an ϵ-stationary point is reached after 𝒪(σ^2ϵ^-2+ τ_maxϵ^-1) iterations, where σ denotes the variance of stochastic gradients. In this work (i) we obtain a tighter convergence rate of 𝒪(σ^2ϵ^-2+ √(τ_maxτ_avg)ϵ^-1) without any change in the algorithm where τ_avg is the average delay, which can be significantly smaller than τ_max. We also provide (ii) a simple delay-adaptive learning rate scheme, under which asynchronous SGD achieves a convergence rate of 𝒪(σ^2ϵ^-2+ τ_avgϵ^-1), and does not require any extra hyperparameter tuning nor extra communications. Our result allows to show for the first time that asynchronous SGD is always faster than mini-batch SGD. In addition, (iii) we consider the case of heterogeneous functions motivated by federated learning applications and improve the convergence rate by proving a weaker dependence on the maximum delay compared to prior works. In particular, we show that the heterogeneity term in convergence rate is only affected by the average delay within each worker.
READ FULL TEXT
