Decentralized EM to Learn Gaussian Mixtures from Datasets Distributed by Features

01/24/2022
by   Pedro Valdeira, et al.
Carnegie Mellon University
0

Expectation Maximization (EM) is the standard method to learn Gaussian mixtures. Yet its classic, centralized form is often infeasible, due to privacy concerns and computational and communication bottlenecks. Prior work dealt with data distributed by examples, horizontal partitioning, but we lack a counterpart for data scattered by features, an increasingly common scheme (e.g. user profiling with data from multiple entities). To fill this gap, we provide an EM-based algorithm to fit Gaussian mixtures to Vertically Partitioned data (VP-EM). In federated learning setups, our algorithm matches the centralized EM fitting of Gaussian mixtures constrained to a subspace. In arbitrary communication graphs, consensus averaging allows VP-EM to run on large peer-to-peer networks as an EM approximation. This mismatch comes from consensus error only, which vanishes exponentially fast with the number of consensus rounds. We demonstrate VP-EM on various topologies for both synthetic and real data, evaluating its approximation of centralized EM and seeing that it outperforms the available benchmark.

READ FULL TEXT
06/27/2012

Convergence of the EM Algorithm for Gaussian Mixtures with Unbalanced Mixing Coefficients

The speed of convergence of the Expectation Maximization (EM) algorithm ...
07/08/2019

Comparing EM with GD in Mixture Models of Two Components

The expectation-maximization (EM) algorithm has been widely used in mini...
01/30/2013

Learning Mixtures of DAG Models

We describe computationally efficient methods for learning mixtures in w...
11/19/2021

An Expectation-Maximization Perspective on Federated Learning

Federated learning describes the distributed training of models across m...
08/05/2015

A MAP approach for ℓ_q-norm regularized sparse parameter estimation using the EM algorithm

In this paper, Bayesian parameter estimation through the consideration o...
11/03/2021

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

The Expectation Maximization (EM) algorithm is the default algorithm for...

Please sign up or login with your details

Forgot password? Click here to reset