On the Limits of Learning Representations with Label-Based Supervision

by   Jiaming Song, et al.
Stanford University

Advances in neural network based classifiers have transformed automatic feature learning from a pipe dream of stronger AI to a routine and expected property of practical systems. Since the emergence of AlexNet every winning submission of the ImageNet challenge has employed end-to-end representation learning, and due to the utility of good representations for transfer learning, representation learning has become as an important and distinct task from supervised learning. At present, this distinction is inconsequential, as supervised methods are state-of-the-art in learning transferable representations. But recent work has shown that generative models can also be powerful agents of representation learning. Will the representations learned from these generative methods ever rival the quality of those from their supervised competitors? In this work, we argue in the affirmative, that from an information theoretic perspective, generative models have greater potential for representation learning. Based on several experimentally validated assumptions, we show that supervised learning is upper bounded in its capacity for representation learning in ways that certain generative models, such as Generative Adversarial Networks (GANs) are not. We hope that our analysis will provide a rigorous motivation for further exploration of generative representation learning.


Large Scale Adversarial Representation Learning

Adversarially trained generative models (GANs) have recently achieved co...

Generative Adversarial Networks for Multimodal Representation Learning in Video Hyperlinking

Continuous multimodal representations suitable for multimodal informatio...

Representation Learning for Non-Melanoma Skin Cancer using a Latent Autoencoder

Generative learning is a powerful tool for representation learning, and ...

GeomCA: Geometric Evaluation of Data Representations

Evaluating the quality of learned representations without relying on a d...

Holographic Neural Architectures

Representation learning is at the heart of what makes deep learning effe...

Bayesian representation learning with oracle constraints

Representation learning systems typically rely on massive amounts of lab...

Please sign up or login with your details

Forgot password? Click here to reset