Policy composition in reinforcement learning via multi-objective policy optimization

by   Shruti Mishra, et al.

We enable reinforcement learning agents to learn successful behavior policies by utilizing relevant pre-existing teacher policies. The teacher policies are introduced as objectives, in addition to the task objective, in a multi-objective policy optimization setting. Using the Multi-Objective Maximum a Posteriori Policy Optimization algorithm (Abdolmaleki et al. 2020), we show that teacher policies can help speed up learning, particularly in the absence of shaping rewards. In two domains with continuous observation and action spaces, our agents successfully compose teacher policies in sequence and in parallel, and are also able to further extend the policies of the teachers in order to solve the task. Depending on the specified combination of task and teacher(s), teacher(s) may naturally act to limit the final performance of an agent. The extent to which agents are required to adhere to teacher policies are determined by hyperparameters which determine both the effect of teachers on learning speed and the eventual performance of the agent on the task. In the humanoid domain (Tassa et al. 2018), we also equip agents with the ability to control the selection of teachers. With this ability, agents are able to meaningfully compose from the teacher policies to achieve a superior task reward on the walk task than in cases without access to the teacher policies. We show the resemblance of composed task policies with the corresponding teacher policies through videos.


page 1

page 2

page 3

page 4


Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learning

Many reinforcement-learning researchers treat the reward function as a p...

Guarded Policy Optimization with Imperfect Online Demonstrations

The Teacher-Student Framework (TSF) is a reinforcement learning setting ...

Lexicographic Multi-Objective Reinforcement Learning

In this work we introduce reinforcement learning techniques for solving ...

APIA: An Architecture for Policy-Aware Intentional Agents

This paper introduces the APIA architecture for policy-aware intentional...

Optimizing the Long-Term Behaviour of Deep Reinforcement Learning for Pushing and Grasping

We investigate the "Visual Pushing for Grasping" (VPG) system by Zeng et...

The Importance of Credo in Multiagent Learning

We propose a model for multi-objective optimization, a credo, for agents...

Psychotherapy AI Companion with Reinforcement Learning Recommendations and Interpretable Policy Dynamics

We introduce a Reinforcement Learning Psychotherapy AI Companion that ge...

Please sign up or login with your details

Forgot password? Click here to reset