Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling

by   Adrian Šošić, et al.

Recent advances in the field of inverse reinforcement learning (IRL) have yielded sophisticated frameworks which relax the original modeling assumption that the behavior of an observed agent reflects only a single intention. Instead, the demonstration data is typically divided into parts, to account for the fact that different trajectories may correspond to different intentions, e.g., because they were generated by different domain experts. In this work, we go one step further: using the intuitive concept of subgoals, we build upon the premise that even a single trajectory can be explained more efficiently locally within a certain context than globally, enabling a more compact representation of the observed behavior. Based on this assumption, we build an implicit intentional model of the agent's goals to forecast its behavior in unobserved situations. The result is an integrated Bayesian prediction framework which provides smooth policy estimates that are consistent with the expert's plan and significantly outperform existing IRL solutions. Most notably, our framework naturally handles situations where the intentions of the agent change with time and classical IRL algorithms fail. In addition, due to its probabilistic nature, the model can be straightforwardly applied in an active learning setting to guide the demonstration process of the expert.


page 14

page 25

page 28

page 30

page 34

page 36


Active Learning for Risk-Sensitive Inverse Reinforcement Learning

One typical assumption in inverse reinforcement learning (IRL) is that h...

RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration

Imitation learning has long been an approach to alleviate the tractabili...

Active Deep Q-learning with Demonstration

Recent research has shown that although Reinforcement Learning (RL) can ...

Neuroevolution-Based Inverse Reinforcement Learning

The problem of Learning from Demonstration is targeted at learning to pe...

Multi-class Generalized Binary Search for Active Inverse Reinforcement Learning

This paper addresses the problem of learning a task from demonstration. ...

Bayesian multitask inverse reinforcement learning

We generalise the problem of inverse reinforcement learning to multiple ...

Blessing from Experts: Super Reinforcement Learning in Confounded Environments

We introduce super reinforcement learning in the batch setting, which ta...

Please sign up or login with your details

Forgot password? Click here to reset