Internal representation dynamics and geometry in recurrent neural networks

01/09/2020
by   Stefan Horoi, et al.
0

The efficiency of recurrent neural networks (RNNs) in dealing with sequential data has long been established. However, unlike deep, and convolution networks where we can attribute the recognition of a certain feature to every layer, it is unclear what "sub-task" a single recurrent step or layer accomplishes. Our work seeks to shed light onto how a vanilla RNN implements a simple classification task by analysing the dynamics of the network and the geometric properties of its hidden states. We find that early internal representations are evocative of the real labels of the data but this information is not directly accessible to the output layer. Furthermore the network's dynamics and the sequence length are both critical to correct classifications even when there is no additional task relevant information provided.

READ FULL TEXT

page 1

page 2

page 3

research
05/22/2018

State-Denoised Recurrent Neural Networks

Recurrent neural networks (RNNs) are difficult to train on sequence proc...
research
02/19/2019

Understanding and Controlling Memory in Recurrent Neural Networks

To be effective in sequential data processing, Recurrent Neural Networks...
research
06/12/2018

Quaternion Recurrent Neural Networks

Recurrent neural networks (RNNs) are powerful architectures to model seq...
research
03/10/2022

Robustness Analysis of Classification Using Recurrent Neural Networks with Perturbed Sequential Input

For a given stable recurrent neural network (RNN) that is trained to per...
research
11/28/2017

Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure

We investigate how neural networks can learn and process languages with ...
research
12/16/2022

Preventing RNN from Using Sequence Length as a Feature

Recurrent neural networks are deep learning topologies that can be train...
research
04/13/2018

Neural Trajectory Analysis of Recurrent Neural Network In Handwriting Synthesis

Recurrent neural networks (RNNs) are capable of learning to generate hig...

Please sign up or login with your details

Forgot password? Click here to reset