Geometry of Linear Convolutional Networks

by   Kathlén Kohn, et al.

We study the family of functions that are represented by a linear convolutional neural network (LCN). These functions form a semi-algebraic subset of the set of linear maps from input space to output space. In contrast, the families of functions represented by fully-connected linear networks form algebraic sets. We observe that the functions represented by LCNs can be identified with polynomials that admit certain factorizations, and we use this perspective to describe the impact of the network's architecture on the geometry of the resulting function space. We further study the optimization of an objective function over an LCN, analyzing critical points in function space and in parameter space, and describing dynamical invariants for gradient descent. Overall, our theory predicts that the optimized parameters of an LCN will often correspond to repeated filters across layers, or filters that can be decomposed as repeated filters. We also conduct numerical and symbolic experiments that illustrate our results and present an in-depth analysis of the landscape for small architectures.


Function Space and Critical Points of Linear Convolutional Networks

We study the geometry of linear networks with one-dimensional convolutio...

Pure and Spurious Critical Points: a Geometric Study of Linear Networks

The critical locus of the loss function of a neural network is determine...

Implicit Bias of Gradient Descent on Linear Convolutional Networks

We show that gradient descent on full-width linear convolutional network...

Convolutional Neural Networks as 2-D systems

This paper introduces a novel representation of convolutional Neural Net...

Theory of Deep Convolutional Neural Networks II: Spherical Analysis

Deep learning based on deep neural networks of various structures and ar...

GMD functions for scheme-based linear codes and algebraic invariants of Geramita ideals

Motivated by notions from coding theory, we study the generalized minimu...

Ultrafast learning of 4-node hybridization cycles in phylogenetic networks using algebraic invariants

The abundance of gene flow in the Tree of Life challenges the notion tha...

Please sign up or login with your details

Forgot password? Click here to reset