A Deeper Look at Discounting Mismatch in Actor-Critic Algorithms

10/02/2020
by   Shangtong Zhang, et al.
1

We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a γ^t term in the actor update for the transition observed at time t in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting (γ^t) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective (γ = 1) where γ^t disappears naturally (1^t = 1). We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective (γ < 1) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.

READ FULL TEXT

page 1

page 2

page 3

page 4

12/19/2018

TD-Regularized Actor-Critic Methods

Actor-critic methods can achieve incredible performance on difficult rei...
03/04/2022

A Small Gain Analysis of Single Timescale Actor Critic

We consider a version of actor-critic which uses proportional step-sizes...
10/06/2021

Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective

Off-policy Actor-Critic algorithms have demonstrated phenomenal experime...
11/20/2018

Representation Learning of Pedestrian Trajectories Using Actor-Critic Sequence-to-Sequence Autoencoder

Representation learning of pedestrian trajectories transforms variable-l...
06/28/2017

An Actor-Critic Contextual Bandit Algorithm for Personalized Mobile Health Interventions

Increasing technological sophistication and widespread use of smartphone...
06/19/2020

Band-limited Soft Actor Critic Model

Soft Actor Critic (SAC) algorithms show remarkable performance in comple...
09/10/2018

Expert-augmented actor-critic for ViZDoom and Montezumas Revenge

We propose an expert-augmented actor-critic algorithm, which we evaluate...

Please sign up or login with your details

Forgot password? Click here to reset