Multi-Environment Pretraining Enables Transfer to Action Limited Datasets

by   David Venuto, et al.
McGill University

Using massive datasets to train large-scale models has emerged as a dominant approach for broad generalization in natural language and vision applications. In reinforcement learning, however, a key challenge is that available data of sequential decision making is often not annotated with actions - for example, videos of game-play are much more available than sequences of frames paired with their logged game controls. We propose to circumvent this challenge by combining large but sparsely-annotated datasets from a target environment of interest with fully-annotated datasets from various other source environments. Our method, Action Limited PreTraining (ALPT), leverages the generalization capabilities of inverse dynamics modelling (IDM) to label missing action data in the target environment. We show that utilizing even one additional environment dataset of labelled data during IDM pretraining gives rise to substantial improvements in generating action labels for unannotated sequences. We evaluate our method on benchmark game-playing environments and show that we can significantly improve game performance and generalization capability compared to other approaches, using annotated datasets equivalent to only 12 minutes of gameplay. Highlighting the power of IDM, we show that these benefits remain even when target and source environments share no common actions.


page 1

page 2

page 3

page 4


Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

Pretraining on noisy, internet-scale datasets has been heavily studied a...

Ego-Only: Egocentric Action Detection without Exocentric Pretraining

We present Ego-Only, the first training pipeline that enables state-of-t...

Plotting Behind the Scenes: Towards Learnable Game Engines

Game engines are powerful tools in computer graphics. Their power comes ...

Masked Autoencoding for Scalable and Generalizable Decision Making

We are interested in learning scalable agents for reinforcement learning...

Vid2Act: Activate Offline Videos for Visual RL

Pretraining RL models on offline video datasets is a promising way to im...

Primitive3D: 3D Object Dataset Synthesis from Randomly Assembled Primitives

Numerous advancements in deep learning can be attributed to the access t...

Hyperspectral Benchmark: Bridging the Gap between HSI Applications through Comprehensive Dataset and Pretraining

Hyperspectral Imaging (HSI) serves as a non-destructive spatial spectros...

Please sign up or login with your details

Forgot password? Click here to reset