Shallow Updates for Deep Reinforcement Learning

by   Nir Levine, et al.

Deep reinforcement learning (DRL) methods such as the Deep Q-Network (DQN) have achieved state-of-the-art results in a variety of challenging, high-dimensional domains. This success is mainly attributed to the power of deep neural networks to learn rich domain representations for approximating the value function or policy. Batch reinforcement learning methods with linear representations, on the other hand, are more stable and require less hyper parameter tuning. Yet, substantial feature engineering is necessary to achieve good results. In this work we propose a hybrid approach -- the Least Squares Deep Q-Network (LS-DQN), which combines rich feature representations learned by a DRL algorithm with the stability of a linear least squares method. We do this by periodically re-training the last hidden layer of a DRL network with a batch least squares update. Key to our approach is a Bayesian regularization term for the least squares update, which prevents over-fitting to the more recent data. We tested LS-DQN on five Atari games and demonstrate significant improvement over vanilla DQN and Double-DQN. We also investigated the reasons for the superior performance of our method. Interestingly, we found that the performance improvement can be attributed to the large batch size used by the LS method when optimizing the last layer.


page 1

page 2

page 3

page 4


Randomized Policy Learning for Continuous State and Action MDPs

Deep reinforcement learning methods have achieved state-of-the-art resul...

Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms

This paper makes one step forward towards characterizing a new family of...

Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks

Reinforcement learning systems require good representations to work well...

Deep Reinforcement Learning with Decorrelation

Learning an effective representation for high-dimensional data is a chal...

Using Generative Adversarial Nets on Atari Games for Feature Extraction in Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) has been successfully applied in sever...

Deep Reinforcement Learning: Framework, Applications, and Embedded Implementations

The recent breakthroughs of deep reinforcement learning (DRL) technique ...

An Exploration of Deep Learning Methods in Hungry Geese

Hungry Geese is a n-player variation of the popular game snake. This pap...

Please sign up or login with your details

Forgot password? Click here to reset