Spectral Ergodicity in Deep Learning Architectures via Surrogate Random Matrices

04/25/2017
by   Mehmet Süzen, et al.
Universitat de les Illes Balears
physics.org
University of Hamburg
0

In this work a novel method to quantify spectral ergodicity for random matrices is presented. The new methodology combines approaches rooted in the metrics of Thirumalai-Mountain (TM) and Kullbach-Leibler (KL) divergence. The method is applied to a general study of deep and recurrent neural networks via the analysis of random matrix ensembles mimicking typical weight matrices of those systems. In particular, we examine circular random matrix ensembles: circular unitary ensemble (CUE), circular orthogonal ensemble (COE), and circular symplectic ensemble (CSE). Eigenvalue spectra and spectral ergodicity are computed for those ensembles as a function of network size. It is observed that as the matrix size increases the level of spectral ergodicity of the ensemble rises, i.e., the eigenvalue spectra obtained for a single realisation at random from the ensemble is closer to the spectra obtained averaging over the whole ensemble. Based on previous results we conjecture that success of deep learning architectures is strongly bound to the concept of spectral ergodicity. The method to compute spectral ergodicity proposed in this work could be used to optimise the size and architecture of deep as well as recurrent neural networks.

READ FULL TEXT

page 1

page 2

page 3

06/13/2020

Beyond Random Matrix Theory for Deep Networks

We investigate whether the Wigner semi-circle and Marcenko-Pastur distri...
06/27/2018

von Mises Tapering: A Circular Data Windowing

Continuous standard windowing is revisited and a new taper shape is intr...
07/17/2019

Distribution of the ratio of two consecutive level spacings in orthogonal to unitary crossover ensembles

The ratio of two consecutive level spacings has emerged as a very useful...
03/04/2020

Fast sampling from β-ensembles

We study sampling algorithms for β-ensembles with time complexity less t...
04/23/2015

Use of Ensembles of Fourier Spectra in Capturing Recurrent Concepts in Data Streams

In this research, we apply ensembles of Fourier encoded spectra to captu...
02/12/2021

Applicability of Random Matrix Theory in Deep Learning

We investigate the local spectral statistics of the loss surface Hessian...
01/19/2021

Householder Dice: A Matrix-Free Algorithm for Simulating Dynamics on Gaussian and Random Orthogonal Ensembles

This paper proposes a new algorithm, named Householder Dice (HD), for si...

Please sign up or login with your details

Forgot password? Click here to reset