Hallucinated Adversarial Control for Conservative Offline Policy Evaluation

by   Jonas Rothfuss, et al.

We study the problem of conservative off-policy evaluation (COPE) where given an offline dataset of environment interactions, collected by other agents, we seek to obtain a (tight) lower bound on a policy's performance. This is crucial when deciding whether a given policy satisfies certain minimal performance/safety criteria before it can be deployed in the real world. To this end, we introduce HAMBO, which builds on an uncertainty-aware learned model of the transition dynamics. To form a conservative estimate of the policy's performance, HAMBO hallucinates worst-case trajectories that the policy may take, within the margin of the models' epistemic confidence regions. We prove that the resulting COPE estimates are valid lower bounds, and, under regularity conditions, show their convergence to the true expected return. Finally, we discuss scalable variants of our approach based on Bayesian Neural Networks and empirically demonstrate that they yield reliable and tight lower bounds in various continuous control environments.


page 1

page 2

page 3

page 4


Bootstrapping with Models: Confidence Intervals for Off-Policy Evaluation

For an autonomous agent, executing a poor policy may be costly or even d...

Model-Free and Model-Based Policy Evaluation when Causality is Uncertain

When decision-makers can directly intervene, policy evaluation algorithm...

Conservative Q-Learning for Offline Reinforcement Learning

Effectively leveraging large, previously collected datasets in reinforce...

Confidence-Conditioned Value Functions for Offline Reinforcement Learning

Offline reinforcement learning (RL) promises the ability to learn effect...

A Unified Framework for Conservative Exploration

We study bandits and reinforcement learning (RL) subject to a conservati...

Kernel Conditional Moment Constraints for Confounding Robust Inference

We study policy evaluation of offline contextual bandits subject to unob...

Certification of Iterative Predictions in Bayesian Neural Networks

We consider the problem of computing reach-avoid probabilities for itera...

Please sign up or login with your details

Forgot password? Click here to reset