Learning Generalizable Representations for Reinforcement Learning via Adaptive Meta-learner of Behavioral Similarities

by   Jianda Chen, et al.

How to learn an effective reinforcement learning-based model for control tasks from high-level visual observations is a practical and challenging problem. A key to solving this problem is to learn low-dimensional state representations from observations, from which an effective policy can be learned. In order to boost the learning of state encoding, recent works are focused on capturing behavioral similarities between state representations or applying data augmentation on visual observations. In this paper, we propose a novel meta-learner-based framework for representation learning regarding behavioral similarities for reinforcement learning. Specifically, our framework encodes the high-dimensional observations into two decomposed embeddings regarding reward and dynamics in a Markov Decision Process (MDP). A pair of meta-learners are developed, one of which quantifies the reward similarity and the other quantifies dynamics similarity over the correspondingly decomposed embeddings. The meta-learners are self-learned to update the state embeddings by approximating two disjoint terms in on-policy bisimulation metric. To incorporate the reward and dynamics terms, we further develop a strategy to adaptively balance their impacts based on different tasks or environments. We empirically demonstrate that our proposed framework outperforms state-of-the-art baselines on several benchmarks, including conventional DM Control Suite, Distracting DM Control Suite and a self-driving task CARLA.


page 7

page 9


Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning

Reinforcement learning methods trained on few environments rarely learn ...

Accelerating Representation Learning with View-Consistent Dynamics in Data-Efficient Reinforcement Learning

Learning informative representations from image-based observations is of...

Temporal-adaptive Hierarchical Reinforcement Learning

Hierarchical reinforcement learning (HRL) helps address large-scale and ...

Goal-oriented inference of environment from redundant observations

The agent learns to organize decision behavior to achieve a behavioral g...

Learning Markov State Abstractions for Deep Reinforcement Learning

The fundamental assumption of reinforcement learning in Markov decision ...

Towards QD-suite: developing a set of benchmarks for Quality-Diversity algorithms

While the field of Quality-Diversity (QD) has grown into a distinct bran...

Efficient Embedding of Semantic Similarity in Control Policies via Entangled Bisimulation

Learning generalizeable policies from visual input in the presence of vi...

Please sign up or login with your details

Forgot password? Click here to reset