Model-Based Reinforcement Learning with SINDy

by   Rushiv Arora, et al.

We draw on the latest advancements in the physics community to propose a novel method for discovering the governing non-linear dynamics of physical systems in reinforcement learning (RL). We establish that this method is capable of discovering the underlying dynamics using significantly fewer trajectories (as little as one rollout with ≤ 30 time steps) than state of the art model learning algorithms. Further, the technique learns a model that is accurate enough to induce near-optimal policies given significantly fewer trajectories than those required by model-free algorithms. It brings the benefits of model-based RL without requiring a model to be developed in advance, for systems that have physics-based dynamics. To establish the validity and applicability of this algorithm, we conduct experiments on four classic control tasks. We found that an optimal policy trained on the discovered dynamics of the underlying system can generalize well. Further, the learned policy performs well when deployed on the actual physical system, thus bridging the model to real system gap. We further compare our method to state-of-the-art model-based and model-free approaches, and show that our method requires fewer trajectories sampled on the true physical system compared other methods. Additionally, we explored approximate dynamics models and found that they also can perform well.


page 1

page 2

page 3

page 4


Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models

Model-based reinforcement learning (RL) algorithms can attain excellent ...

Physical Derivatives: Computing policy gradients by physical forward-propagation

Model-free and model-based reinforcement learning are two ends of a spec...

Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning

Model-based reinforcement learning (RL) is more sample efficient than mo...

Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements

In this paper, we propose a derivative-free model learning framework for...

Accelerating Goal-Directed Reinforcement Learning by Model Characterization

We propose a hybrid approach aimed at improving the sample efficiency in...

Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

A limitation of model-based reinforcement learning (MBRL) is the exploit...

FLEX: an Adaptive Exploration Algorithm for Nonlinear Systems

Model-based reinforcement learning is a powerful tool, but collecting da...

Please sign up or login with your details

Forgot password? Click here to reset