Active Preference-Based Gaussian Process Regression for Reward Learning

by   Erdem Bıyık, et al.

Designing reward functions is a challenging problem in AI and robotics. Humans usually have a difficult time directly specifying all the desirable behaviors that a robot needs to optimize. One common approach is to learn reward functions from collected expert demonstrations. However, learning reward functions from demonstrations introduces many challenges ranging from methods that require highly structured models, e.g. reward functions that are linear in some predefined set of features to less structured reward functions that on the other hand require tremendous amount of data. In addition, humans tend to have a difficult time providing demonstrations on robots with high degrees of freedom, or even quantifying reward values for given demonstrations. To address these challenges, we present a preference-based learning approach, where as an alternative, the human feedback is only of the form of comparisons between trajectories. Furthermore, we do not assume highly constrained structures on the reward function. Instead, we model the reward function using a Gaussian Process (GP) and propose a mathematical formulation to actively find a GP using only human preferences. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework. Our results in simulations and a user study suggest that our approach can efficiently learn expressive reward functions for robotics tasks.


page 2

page 3

page 4

page 5

page 6

page 7

page 9

page 10


Learning Reward Functions by Integrating Human Demonstrations and Preferences

Our goal is to accurately and efficiently learn reward functions for aut...

Learning Preferences for Interactive Autonomy

When robots enter everyday human environments, they need to understand t...

Learning Multimodal Rewards from Rankings

Learning from human feedback has shown to be a useful approach in acquir...

Preference-based Learning of Reward Function Features

Preference-based learning of reward functions, where the reward function...

Asking Easy Questions: A User-Friendly Approach to Active Reward Learning

Robots can learn the right reward function by querying a human expert. E...

Learning from Outside the Viability Kernel: Why we Should Build Robots that can Fall with Grace

Despite impressive results using reinforcement learning to solve complex...

Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences

Reward functions are a common way to specify the objective of a robot. A...

Please sign up or login with your details

Forgot password? Click here to reset