Discovering Blind Spots in Reinforcement Learning

by   Ramya Ramakrishnan, et al.

Agents trained in simulation may make errors in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult to discover because the agent cannot predict them a priori. We propose using oracle feedback to learn a predictive model of these blind spots to reduce costly errors in real-world applications. We focus on blind spots in reinforcement learning (RL) that occur due to incomplete state representation: The agent does not have the appropriate features to represent the true state of the world and thus cannot distinguish among numerous states. We formalize the problem of discovering blind spots in RL as a noisy supervised learning problem with class imbalance. We learn models to predict blind spots in unseen regions of the state space by combining techniques for label aggregation, calibration, and supervised learning. The models take into consideration noise emerging from different forms of oracle feedback, including demonstrations and corrections. We evaluate our approach on two domains and show that it achieves higher predictive performance than baseline methods, and that the learned model can be used to selectively query an oracle at execution time to prevent errors. We also empirically analyze the biases of various feedback types and how they influence the discovery of blind spots.


Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration

Predictive models deployed in the real world may assign incorrect labels...

A Bayesian Approach to Identifying Representational Errors

Trained AI systems and expert decision makers can make errors that are o...

Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning

Accurate and precise terrain estimation is a difficult problem for robot...

Tackling Unbounded State Spaces in Continuing Task Reinforcement Learning

While deep reinforcement learning (RL) algorithms have been successfully...

Provable Safe Reinforcement Learning with Binary Feedback

Safety is a crucial necessity in many applications of reinforcement lear...

On the Theory of Reinforcement Learning with Once-per-Episode Feedback

We study a theory of reinforcement learning (RL) in which the learner re...

Runaway Feedback Loops in Predictive Policing

Predictive policing systems are increasingly used to determine how to al...

Please sign up or login with your details

Forgot password? Click here to reset