Topological Neural Discrete Representation Learning à la Kohonen

by   Kazuki Irie, et al.

Unsupervised learning of discrete representations from continuous ones in neural networks (NNs) is the cornerstone of several applications today. Vector Quantisation (VQ) has become a popular method to achieve such representations, in particular in the context of generative models such as Variational Auto-Encoders (VAEs). For example, the exponential moving average-based VQ (EMA-VQ) algorithm is often used. Here we study an alternative VQ algorithm based on the learning rule of Kohonen Self-Organising Maps (KSOMs; 1982) of which EMA-VQ is a special case. In fact, KSOM is a classic VQ algorithm which is known to offer two potential benefits over the latter: empirically, KSOM is known to perform faster VQ, and discrete representations learned by KSOM form a topological structure on the grid whose nodes are the discrete symbols, resulting in an artificial version of the topographic map in the brain. We revisit these properties by using KSOM in VQ-VAEs for image processing. In particular, our experiments show that, while the speed-up compared to well-configured EMA-VQ is only observable at the beginning of training, KSOM is generally much more robust than EMA-VQ, e.g., w.r.t. the choice of initialisation schemes. Our code is public.


page 3

page 7

page 8

page 14

page 15


Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders

Unsupervised representation learning of speech has been of keen interest...

Local distance preserving auto-encoders using Continuous k-Nearest Neighbours graphs

Auto-encoder models that preserve similarities in the data are a popular...

GeomCA: Geometric Evaluation of Data Representations

Evaluating the quality of learned representations without relying on a d...

Is Information in the Brain Represented in Continuous or Discrete Form?

The question of continuous-versus-discrete information representation in...

Discrete and continuous representations and processing in deep learning: Looking forward

Discrete and continuous representations of content (e.g., of language or...

Extraction of topological features from communication network topological patterns using self-organizing feature maps

Different classes of communication network topologies and their represen...

Please sign up or login with your details

Forgot password? Click here to reset