What Do Neural Networks Learn When Trained With Random Labels?

by   Hartmut Maennel, et al.

We study deep neural networks (DNNs) trained on natural image data with entirely random labels. Despite its popularity in the literature, where it is often used to study memorization, generalization, and other phenomena, little is known about what DNNs learn in this setting. In this paper, we show analytically for convolutional and fully connected networks that an alignment between the principal components of network parameters and data takes place when training with random labels. We study this alignment effect by investigating neural networks pre-trained on randomly labelled image data and subsequently fine-tuned on disjoint datasets with random or real labels. We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch even after accounting for simple effects, such as weight scaling. We analyze how competing effects, such as specialization at later layers, may hide the positive transfer. These effects are studied in several network architectures, including VGG16 and ResNet18, on CIFAR10 and ImageNet.


page 1

page 2

page 3

page 4


Neural Network Memorization Dissection

Deep neural networks (DNNs) can easily fit a random labeling of the trai...

A Light in the Dark: Deep Learning Practices for Industrial Computer Vision

In recent years, large pre-trained deep neural networks (DNNs) have revo...

Detecting Learning vs Memorization in Deep Neural Networks using Shared Structure Validation Sets

The roles played by learning and memorization represent an important top...

Understanding the Benefits of Image Augmentations

Image Augmentations are widely used to reduce overfitting in neural netw...

Directing DNNs Attention for Facial Attribution Classification using Gradient-weighted Class Activation Mapping

Deep neural networks (DNNs) have a high accuracy on image classification...

Knowledge transfer in deep block-modular neural networks

Although deep neural networks (DNNs) have demonstrated impressive result...

Plugin Networks for Inference under Partial Evidence

In this paper, we propose a novel method to incorporate partial evidence...

Please sign up or login with your details

Forgot password? Click here to reset