A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks

by   Ali Gorji, et al.

Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards “simpler” functions. Various notions of simplicity have been introduced to characterize this behavior. Here, we focus on the case of neural networks with discrete (zero-one) inputs through the lens of their Fourier (Walsh-Hadamard) transforms, where the notion of simplicity can be captured through the degree of the Fourier coefficients. We empirically show that neural networks have a tendency to learn lower-degree frequencies. We show how this spectral bias towards simpler features can in fact hurt the neural network's generalization on real-world datasets. To remedy this we propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies. Our regularizer also helps avoid erroneous identification of low-degree frequencies, which further improves generalization. We extensively evaluate our regularizer on synthetic datasets to gain insights into its behavior. Finally, we show significantly improved generalization on four different datasets compared to standard neural networks and other relevant baselines.


page 4

page 19


Spectral Bias in Practice: The Role of Function Frequency in Generalization

Despite their ability to represent highly expressive functions, deep lea...

How You Start Matters for Generalization

Characterizing the remarkable generalization properties of over-paramete...

Simplicity Bias in 1-Hidden Layer Neural Networks

Recent works have demonstrated that neural networks exhibit extreme simp...

The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies

We study the relationship between the speed at which a neural network le...

Fourier Neural Networks: A Comparative Study

We review neural network architectures which were motivated by Fourier s...

Understanding robustness and generalization of artificial neural networks through Fourier masks

Despite the enormous success of artificial neural networks (ANNs) in man...

Constructing Extreme Learning Machines with zero Spectral Bias

The phenomena of Spectral Bias, where the higher frequency components of...

Please sign up or login with your details

Forgot password? Click here to reset