Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics
share
Recently, multi-layer perceptrons (MLPs) with ReLU activations have enabled new photo-realistic rendering techniques by encoding scene properties using their weights. For these models, termed coordinate based MLPs, sinusoidal encodings are necessary in allowing for convergence to the high frequency components of the signal due to their severe spectral bias. Previous work has explained this phenomenon using Neural Tangent Kernel (NTK) and Fourier analysis. However, the kernel regime does not expose the properties of the network that induce this behavior, and the Fourier decomposition is global, not allowing for insight on the network's local dynamics. A new interpretation of spectral bias directly through ReLU network computations would expose their limitations in dense settings, while providing a clearer explanation as to how this behavior emerges during the learning process. In this paper, we provide the first study of spectral bias in a coordinate based MLP through its activation regions and gradient descent dynamics, specifically using gradient confusion. We relate the confusion between inputs to the distinctiveness of their activation patterns, and find higher amounts of confusion when expressive power is limited. This leads to slower convergence to the high frequency components of the signal, which is magnified by the density of coordinates. Additionally, this method allows us to analyze the properties of the activation regions as spectral bias is reduced, in which we find distinct dynamics.
READ FULL TEXT