Neural networks learn to magnify areas near decision boundaries

01/26/2023

∙

We study how training molds the Riemannian geometry induced by neural network feature maps. At infinite width, neural networks with random parameters induce highly symmetric metrics on input space. Feature learning in networks trained to perform classification tasks magnifies local areas along decision boundaries. These changes are consistent with previously proposed geometric approaches for hand-tuning of kernel methods to improve generalization.

READ FULL TEXT

Neural networks learn to magnify areas near decision boundaries

On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective

Feature Learning in Infinite-Width Neural Networks

Unsupervised learning of features and object boundaries from local prediction

Loss Function Entropy Regularization for Diverse Decision Boundaries

The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective

Neural networks learn to magnify areas near decision boundaries

Related Research

On the Decision Boundaries of Deep Neural Networks: A Tropical Geometry Perspective

Feature Learning in Infinite-Width Neural Networks

Unsupervised learning of features and object boundaries from local prediction

Loss Function Entropy Regularization for Diverse Decision Boundaries

The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks

Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks

Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspective