Quantifying and maximizing the information flux in recurrent neural networks

by   Claus Metzner, et al.

Free-running Recurrent Neural Networks (RNNs), especially probabilistic models, generate an ongoing information flux that can be quantified with the mutual information I[x⃗(t),x⃗(t+1)] between subsequent system states x⃗. Although, former studies have shown that I depends on the statistics of the network's connection weights, it is unclear (1) how to maximize I systematically and (2) how to quantify the flux in large systems where computing the mutual information becomes intractable. Here, we address these questions using Boltzmann machines as model systems. We find that in networks with moderately strong connections, the mutual information I is approximately a monotonic transformation of the root-mean-square averaged Pearson correlations between neuron-pairs, a quantity that can be efficiently computed even in large systems. Furthermore, evolutionary maximization of I[x⃗(t),x⃗(t+1)] reveals a general design principle for the weight matrices enabling the systematic construction of systems with a high spontaneous information flux. Finally, we simultaneously maximize information flux and the mean period length of cyclic attractors in the state space of these dynamical networks. Our results are potentially useful for the construction of RNNs that serve as short-time memories or pattern generators.


page 19

page 20

page 21


Mutual Information Scaling and Expressive Power of Sequence Models

Sequence models assign probabilities to variable-length sequences such a...

Use of recurrent infomax to improve the memory capability of input-driven recurrent neural networks

The inherent transient dynamics of recurrent neural networks (RNNs) have...

Mutual Information Decay Curves and Hyper-Parameter Grid Search Design for Recurrent Neural Architectures

We present an approach to design the grid searches for hyper-parameter o...

Criticality in Formal Languages and Statistical Physics

We show that the mutual information between two symbols, as a function o...

A Biologically Plausible Learning Rule for Perceptual Systems of organisms that Maximize Mutual Information

It is widely believed that the perceptual system of an organism is optim...

APS: Active Pretraining with Successor Features

We introduce a new unsupervised pretraining objective for reinforcement ...

Please sign up or login with your details

Forgot password? Click here to reset