DeepAI AI Chat
Log In Sign Up

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

11/03/2021
by   Aymeric Dieuleveut, et al.
0

The Expectation Maximization (EM) algorithm is the default algorithm for inference in latent variable models. As in any other field of machine learning, applications of latent variable models to very large datasets make the use of advanced parallel and distributed architectures mandatory. This paper introduces FedEM, which is the first extension of the EM algorithm to the federated learning context. FedEM is a new communication efficient method, which handles partial participation of local devices, and is robust to heterogeneous distributions of the datasets. To alleviate the communication bottleneck, FedEM compresses appropriately defined complete data sufficient statistics. We also develop and analyze an extension of FedEM to further incorporate a variance reduction scheme. In all cases, we derive finite-time complexity bounds for smooth non-convex problems. Numerical results are presented to support our theoretical findings, as well as an application to federated missing values imputation for biodiversity monitoring.

READ FULL TEXT

page 1

page 2

page 3

page 4

11/24/2020

Geom-SPIDER-EM: Faster Variance Reduced Stochastic Expectation Maximization for Nonconvex Finite-Sum Optimization

The Expectation Maximization (EM) algorithm is a key reference for infer...
11/30/2020

A Stochastic Path-Integrated Differential EstimatoR Expectation Maximization Algorithm

The Expectation Maximization (EM) algorithm is of key importance for inf...
11/19/2021

An Expectation-Maximization Perspective on Federated Learning

Federated learning describes the distributed training of models across m...
08/22/2023

EM for Mixture of Linear Regression with Clustered Data

Modern data-driven and distributed learning frameworks deal with diverse...
05/18/2009

A statistical learning approach to color demosaicing

A statistical learning/inference framework for color demosaicing is pres...
01/24/2022

Decentralized EM to Learn Gaussian Mixtures from Datasets Distributed by Features

Expectation Maximization (EM) is the standard method to learn Gaussian m...