Neural Mixture Models with Expectation-Maximization for End-to-end Deep Clustering

by   Dumindu Tissera, et al.

Any clustering algorithm must synchronously learn to model the clusters and allocate data to those clusters in the absence of labels. Mixture model-based methods model clusters with pre-defined statistical distributions and allocate data to those clusters based on the cluster likelihoods. They iteratively refine those distribution parameters and member assignments following the Expectation-Maximization (EM) algorithm. However, the cluster representability of such hand-designed distributions that employ a limited amount of parameters is not adequate for most real-world clustering tasks. In this paper, we realize mixture model-based clustering with a neural network where the final layer neurons, with the aid of an additional transformation, approximate cluster distribution outputs. The network parameters pose as the parameters of those distributions. The result is an elegant, much-generalized representation of clusters than a restricted mixture of hand-designed distributions. We train the network end-to-end via batch-wise EM iterations where the forward pass acts as the E-step and the backward pass acts as the M-step. In image clustering, the mixture-based EM objective can be used as the clustering objective along with existing representation learning methods. In particular, we show that when mixture-EM optimization is fused with consistency optimization, it improves the sole consistency optimization performance in clustering. Our trained networks outperform single-stage deep clustering methods that still depend on k-means, with unsupervised classification accuracy of 63.8 25.9


page 12

page 16

page 17


An Improved EM algorithm

In this paper, we firstly give a brief introduction of expectation maxim...

Generative Mixture of Networks

A generative model based on training deep architectures is proposed. The...

Clustering of check-in sequences using the mixture Markov chain process

This work is devoted to the clustering of check-in sequences from a geos...

Feature screening for clustering analysis

In this paper, we consider feature screening for ultrahigh dimensional c...

Efficient mixture model for clustering of sparse high dimensional binary data

In this paper we propose a mixture model, SparseMix, for clustering of s...

An Information-Theoretic Analysis of Hard and Soft Assignment Methods for Clustering

Assignment methods are at the heart of many algorithms for unsupervised ...

Improving Gradient Flow with Unrolled Highway Expectation Maximization

Integrating model-based machine learning methods into deep neural archit...

Please sign up or login with your details

Forgot password? Click here to reset