Why Target Networks Stabilise Temporal Difference Methods

by   Mattie Fellows, et al.

Integral to recent successes in deep reinforcement learning has been a class of temporal difference methods that use infrequently updated target values for policy evaluation in a Markov Decision Process. Yet a complete theoretical explanation for the effectiveness of target networks remains elusive. In this work, we provide an analysis of this popular class of algorithms, to finally answer the question: `why do target networks stabilise TD learning'? To do so, we formalise the notion of a partially fitted policy evaluation method, which describes the use of target networks and bridges the gap between fitted methods and semigradient temporal difference algorithms. Using this framework we are able to uniquely characterise the so-called deadly triad - the use of TD updates with (nonlinear) function approximation and off-policy data - which often leads to nonconvergent algorithms. This insight leads us to conclude that the use of target networks can mitigate the effects of poor conditioning in the Jacobian of the TD update. Instead, we show that under mild regularity conditions and a well tuned target network update frequency, convergence can be guaranteed even in the extremely challenging off-policy sampling and nonlinear function approximation setting.


page 1

page 2

page 3

page 4


A Convergent Off-Policy Temporal Difference Algorithm

Learning the value function of a given policy (target policy) from the d...

Breaking the Deadly Triad with a Target Network

The deadly triad refers to the instability of a reinforcement learning a...

Target-Based Temporal Difference Learning

The use of target networks has been a popular and key component of recen...

Consistent On-Line Off-Policy Evaluation

The problem of on-line off-policy evaluation (OPE) has been actively stu...

Investigating practical linear temporal difference learning

Off-policy reinforcement learning has many applications including: learn...

Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates

Temporal-Difference (TD) learning methods, such as Q-Learning, have prov...

Periodic Q-Learning

The use of target networks is a common practice in deep reinforcement le...

Please sign up or login with your details

Forgot password? Click here to reset