The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning

by   Andrei Nica, et al.
Montréal Institute of Learning Algorithms
McGill University

Decision-making AI agents are often faced with two important challenges: the depth of the planning horizon, and the branching factor due to having many choices. Hierarchical reinforcement learning methods aim to solve the first problem, by providing shortcuts that skip over multiple time steps. To cope with the breadth, it is desirable to restrict the agent's attention at each step to a reasonable number of possible choices. The concept of affordances (Gibson, 1977) suggests that only certain actions are feasible in certain states. In this work, we model "affordances" through an attention mechanism that limits the available choices of temporally extended options. We present an online, model-free algorithm to learn affordances that can be used to further learn subgoal options. We investigate the role of hard versus soft attention in training data collection, abstract value learning in long-horizon tasks, and handling a growing number of choices. We identify and empirically illustrate the settings in which the paradox of choice arises, i.e. when having fewer but more meaningful choices improves the learning speed and performance of a reinforcement learning agent.


Multi-agent Deep Covering Option Discovery

The use of options can greatly accelerate exploration in reinforcement l...

Learning Temporally Extended Skills in Continuous Domains as Symbolic Actions for Planning

Problems which require both long-horizon planning and continuous control...

Towards Interpretable Reinforcement Learning Using Attention Augmented Agents

Inspired by recent work in attention models for image captioning and que...

Decision-making with E-admissibility given a finite assessment of choices

Given information about which options a decision-maker definitely reject...

A Model of Free Will for Artificial Entities

The impression of free will is the feeling according to which our choice...

Towards an Interpretable Hierarchical Agent Framework using Semantic Goals

Learning to solve long horizon temporally extended tasks with reinforcem...

Temporally Abstract Partial Models

Humans and animals have the ability to reason and make predictions about...

Please sign up or login with your details

Forgot password? Click here to reset