Temporally Abstract Partial Models

by   Khimya Khetarpal, et al.

Humans and animals have the ability to reason and make predictions about different courses of action at many time scales. In reinforcement learning, option models (Sutton, Precup & Singh, 1999; Precup, 2000) provide the framework for this kind of temporally abstract prediction and reasoning. Natural intelligent agents are also able to focus their attention on courses of action that are relevant or feasible in a given situation, sometimes termed affordable actions. In this paper, we define a notion of affordances for options, and develop temporally abstract partial option models, that take into account the fact that an option might be affordable only in certain situations. We analyze the trade-offs between estimation and approximation error in planning and learning when using such models, and identify some interesting special cases. Additionally, we demonstrate empirically the potential impact of partial option models on the efficiency of planning.


page 1

page 2

page 3

page 4


Reward-Respecting Subtasks for Model-Based Reinforcement Learning

To achieve the ambitious goals of artificial intelligence, reinforcement...

Learnings Options End-to-End for Continuous Action Tasks

We present new results on learning temporally extended actions for conti...

MDP Abstraction with Successor Features

Abstraction plays an important role for generalisation of knowledge and ...

Successor Options: An Option Discovery Framework for Reinforcement Learning

The options framework in reinforcement learning models the notion of a s...

Decision-Theoretic Planning with Concurrent Temporally Extended Actions

We investigate a model for planning under uncertainty with temporallyext...

The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning

Decision-making AI agents are often faced with two important challenges:...

A Formal Metareasoning Model of Concurrent Planning and Execution

Agents that plan and act in the real world must deal with the fact that ...

Please sign up or login with your details

Forgot password? Click here to reset