Interference and Generalization in Temporal Difference Learning

by   Emmanuel Bengio, et al.

We study the link between generalization and interference in temporal-difference (TD) learning. Interference is defined as the inner product of two different gradients, representing their alignment. This quantity emerges as being of interest from a variety of observations about neural networks, parameter sharing and the dynamics of learning. We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning. We hypothesize that the cause can be traced back to the interplay between the dynamics of interference and bootstrapping. This is supported empirically by several observations: the negative relationship between the generalization gap and interference in TD, the negative effect of bootstrapping on interference and the local coherence of targets, and the contrast between the propagation rate of information in TD(0) versus TD(λ) and regression tasks such as Monte-Carlo policy evaluation. We hope that these new findings can guide the future discovery of better bootstrapping methods.


On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment

Modern multilingual models are trained on concatenated text from multipl...

Learning Dynamics and Generalization in Reinforcement Learning

Solving a reinforcement learning (RL) problem poses two competing challe...

Auto-Correlation and Coherence Time of Interference in Poisson Networks

The dynamics of interference over space and time influences the performa...

Class Interference of Deep Neural Networks

Recognizing and telling similar objects apart is even hard for human bei...

Slimmable Networks for Contrastive Self-supervised Learning

Self-supervised learning makes great progress in large model pre-trainin...

On Interference of Signals and Generalization in Feedforward Neural Networks

This paper studies how the generalization ability of neurons can be affe...

More Causes Less Effect: Destructive Interference in Decision Making

We present a new experiment demonstrating destructive interference in cu...

Please sign up or login with your details

Forgot password? Click here to reset