Spectral Bias in Practice: The Role of Function Frequency in Generalization

by   Sara Fridovich-Keil, et al.

Despite their ability to represent highly expressive functions, deep learning models trained with SGD seem to find simple, constrained solutions that generalize surprisingly well. Spectral bias - the tendency of neural networks to prioritize learning low frequency functions - is one possible explanation for this phenomenon, but so far spectral bias has only been observed in theoretical models and simplified experiments. In this work, we propose methodologies for measuring spectral bias in modern image classification networks. We find that these networks indeed exhibit spectral bias, and that networks that generalize well strike a balance between having enough complexity(i.e. high frequencies) to fit the data while being simple enough to avoid overfitting. For example, we experimentally show that larger models learn high frequencies faster than smaller ones, but many forms of regularization, both explicit and implicit, amplify spectral bias and delay the learning of high frequencies. We also explore the connections between function frequency and image frequency and find that spectral bias is sensitive to the low frequencies prevalent in natural images. Our work enables measuring and ultimately controlling the spectral behavior of neural networks used for image classification, and is a step towards understanding why deep models generalize well


page 5

page 6


The Spectral Bias of Polynomial Neural Networks

Polynomial neural networks (PNNs) have been recently shown to be particu...

A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks

Despite the capacity of neural nets to learn arbitrary functions, models...

What do neural networks learn in image classification? A frequency shortcut perspective

Frequency analysis is useful for understanding the mechanisms of represe...

Understanding robustness and generalization of artificial neural networks through Fourier masks

Despite the enormous success of artificial neural networks (ANNs) in man...

The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies

We study the relationship between the speed at which a neural network le...

Constructing Extreme Learning Machines with zero Spectral Bias

The phenomena of Spectral Bias, where the higher frequency components of...

Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs

Over-parameterized deep neural networks (DNNs) with sufficient capacity ...

Please sign up or login with your details

Forgot password? Click here to reset