Calibrated Model-Based Deep Reinforcement Learning

by   Ali Malik, et al.

Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that good uncertainties must be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the HalfCheetah MuJoCo task, our system achieves state-of-the-art performance using 50% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.


page 1

page 2

page 3

page 4


Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning

Sample efficiency has been one of the major challenges for deep reinforc...

Accurate Uncertainties for Deep Learning Using Calibrated Regression

Methods for reasoning under uncertainty are a key building block of accu...

Calibrated and Sharp Uncertainties in Deep Learning via Simple Density Estimation

Predictive uncertainties can be characterized by two properties–calibrat...

Calibrating generalized predictive distributions

In prediction problems, it is common to model the data-generating proces...

Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?

We contribute to micro-data model-based reinforcement learning (MBRL) by...

Learning Trajectories for Visual-Inertial System Calibration via Model-based Heuristic Deep Reinforcement Learning

Visual-inertial systems rely on precise calibrations of both camera intr...

On the role of planning in model-based deep reinforcement learning

Model-based planning is often thought to be necessary for deep, careful ...

Please sign up or login with your details

Forgot password? Click here to reset