Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement

by   Andre Barreto, et al.

The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach. Recently, a framework based on two ideas, successor features (SFs) and generalised policy improvement (GPI), has been introduced as a principled way of transferring skills. In this paper we extend the SFs & GPI framework in two ways. One of the basic assumptions underlying the original formulation of SFs & GPI is that rewards for all tasks of interest can be computed as linear combinations of a fixed set of features. We relax this constraint and show that the theoretical guarantees supporting the framework can be extended to any set of tasks that only differ in the reward function. Our second contribution is to show that one can use the reward functions themselves as features for future tasks, without any loss of expressiveness, thus removing the need to specify a set of features beforehand. This makes it possible to combine SFs & GPI with deep learning in a more stable way. We empirically verify this claim on a complex 3D environment where observations are images from a first-person perspective. We show that the transfer promoted by SFs & GPI leads to very good policies on unseen tasks almost instantaneously. We also describe how to learn policies specialised to the new tasks in a way that allows them to be added to the agent's set of skills, and thus be reused in the future.


page 1

page 2

page 3

page 4


Universal Successor Features Approximators

The ability of a reinforcement learning (RL) agent to learn about many r...

Successor Features for Transfer in Reinforcement Learning

Transfer in reinforcement learning refers to the notion that generalizat...

Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates

We study the problem of learning a good set of policies, so that when co...

Task Relabelling for Multi-task Transfer using Successor Features

Deep Reinforcement Learning has been very successful recently with vario...

Successor Feature Neural Episodic Control

A longstanding goal in reinforcement learning is to build intelligent ag...

An Extensible Interactive Interface for Agent Design

In artificial intelligence, we often specify tasks through a reward func...

Entropic Policy Composition with Generalized Policy Improvement and Divergence Correction

Deep reinforcement learning (RL) algorithms have made great strides in r...

Please sign up or login with your details

Forgot password? Click here to reset