Hypernetworks in Meta-Reinforcement Learning

by   Jacob Beck, et al.

Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the art methods often fail to outperform a degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate policies of the degenerate solution while also allowing for generalization across tasks, and are applicable to meta-RL. However, evidence from supervised learning suggests hypernetwork performance is highly sensitive to the initialization. In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, and that naive initializations yield poor performance; 2) propose a novel hypernetwork initialization scheme that matches or exceeds the performance of a state-of-the-art approach proposed for supervised settings, as well as being simpler and more general; and 3) use this method to show that hypernetworks can improve performance in meta-RL by evaluating on multiple simulated robotics benchmarks.


Robust Meta-Reinforcement Learning with Curriculum-Based Task Sampling

Meta-reinforcement learning (meta-RL) acquires meta-policies that show g...

Hindsight Foresight Relabeling for Meta-Reinforcement Learning

Meta-reinforcement learning (meta-RL) algorithms allow for agents to lea...

Pre-training as Batch Meta Reinforcement Learning with tiMe

Pre-training is transformative in supervised learning: a large network t...

Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces

Meta-reinforcement learning (RL) addresses the problem of sample ineffic...

On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning

Intelligent agents should have the ability to leverage knowledge from pr...

Curriculum-based Asymmetric Multi-task Reinforcement Learning

We introduce CAMRL, the first curriculum-based asymmetric multi-task lea...

Robust Subtask Learning for Compositional Generalization

Compositional reinforcement learning is a promising approach for trainin...

Please sign up or login with your details

Forgot password? Click here to reset