Generalizing Point Embeddings using the Wasserstein Space of Elliptical Distributions

by   Boris Muzellec, et al.

Embedding complex objects as vectors in low dimensional spaces is a longstanding problem in machine learning. We propose in this work an extension of that approach, which consists in embedding objects as elliptical probability distributions, namely distributions whose densities have elliptical level sets. We endow these measures with the 2-Wasserstein metric, with two important benefits: (i) For such measures, the squared 2-Wasserstein metric has a closed form, equal to the sum of the squared Euclidean distance between means and the squared Bures metric between covariance matrices. The latter is a Riemannian metric between positive semi-definite matrices, which turns out to be Euclidean on a suitable factor representation of such matrices, which is valid on the entire geodesic between these matrices. (ii) The 2-Wasserstein distance boils down to the usual Euclidean metric when comparing Diracs, and therefore provides the natural framework to extend point embeddings. We show that for these reasons Wasserstein elliptical embeddings are more intuitive and yield tools that are better behaved numerically than the alternative choice of Gaussian embeddings with the Kullback-Leibler divergence. In particular, and unlike previous work based on the KL geometry, we learn elliptical distributions that are not necessarily diagonal. We demonstrate the interest of elliptical embeddings by using them for visualization, to compute embeddings of words, and to reflect entanglement or hypernymy.


page 1

page 2

page 3

page 4


Learning Embeddings into Entropic Wasserstein Spaces

Euclidean embeddings of data are fundamentally limited in their ability ...

Permutation invariant networks to learn Wasserstein metrics

Understanding the space of probability measures on a metric space equipp...

Probability Metrics for Tropical Spaces of Different Dimensions

The problem of comparing probability distributions is at the heart of ma...

Ensemble Riemannian Data Assimilation over the Wasserstein Space

In this paper, we present a new ensemble data assimilation paradigm over...

Wasserstein Training of Boltzmann Machines

The Boltzmann machine provides a useful framework to learn highly comple...

Generating valid Euclidean distance matrices

Generating point clouds, e.g., molecular structures, in arbitrary rotati...

Please sign up or login with your details

Forgot password? Click here to reset