Anchor Transform: Learning Sparse Representations of Discrete Objects

by   Paul Pu Liang, et al.

Learning continuous representations of discrete objects such as text, users, and URLs lies at the heart of many applications including language and user modeling. When using discrete objects as input to neural networks, we often ignore the underlying structures (e.g. natural groupings and similarities) and embed the objects independently into individual vectors. As a result, existing methods do not scale to large vocabulary sizes. In this paper, we design a Bayesian nonparametric prior for embeddings that encourages sparsity and leverages natural groupings among objects. We derive an approximate inference algorithm based on Small Variance Asymptotics which yields a simple and natural algorithm for learning a small set of anchor embeddings and a sparse transformation matrix. We call our method Anchor Transform (ANT) as the embeddings of discrete objects are a sparse linear combination of the anchors, weighted according to the transformation matrix. ANT is scalable, flexible, end-to-end trainable, and allows the user to incorporate domain knowledge about object relationships. On text classification and language modeling benchmarks, ANT demonstrates stronger performance with fewer parameters as compared to existing compression baselines.


page 1

page 2

page 3

page 4


Transformation of Dense and Sparse Text Representations

Sparsity is regarded as a desirable property of representations, especia...

Learning Domain-Specific Word Embeddings from Sparse Cybersecurity Texts

Word embedding is a Natural Language Processing (NLP) technique that aut...

Seq2RDF: An end-to-end application for deriving Triples from Natural Language Text

We present an end-to-end approach that takes unstructured textual input ...

Encoding word order in complex embeddings

Sequential word order is important when processing text. Currently, neur...

Anchor Distance for 3D Multi-Object Distance Estimation from 2D Single Shot

Visual perception of the objects in a 3D environment is a key to success...

Deep Active Learning for Anchor User Prediction

Predicting pairs of anchor users plays an important role in the cross-ne...

Please sign up or login with your details

Forgot password? Click here to reset