Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning

by   Zhibo Yang, et al.

Being able to predict human gaze behavior has obvious importance for behavioral vision and for computer vision applications. Most models have mainly focused on predicting free-viewing behavior using saliency maps, but these predictions do not generalize to goal-directed behavior, such as when a person searches for a visual target object. We propose the first inverse reinforcement learning (IRL) model to learn the internal reward function and policy used by humans during visual search. The viewer's internal belief states were modeled as dynamic contextual belief maps of object locations. These maps were learned by IRL and then used to predict behavioral scanpaths for multiple target categories. To train and evaluate our IRL model we created COCO-Search18, which is now the largest dataset of high-quality search fixations in existence. COCO-Search18 has 10 participants searching for each of 18 target-object categories in 6202 images, making about 300,000 goal-directed fixations. When trained and evaluated on COCO-Search18, the IRL model outperformed baseline models in predicting search fixation scanpaths, both in terms of similarity to human search behavior and search efficiency. Finally, reward maps recovered by the IRL model reveal distinctive target-dependent patterns of object prioritization, which we interpret as a learned object context.


page 1

page 3

page 7

page 8

page 12

page 13

page 14


Predicting Goal-directed Attention Control Using Inverse-Reinforcement Learning

Understanding how goal states control behavior is a question ripe for in...

Target-absent Human Attention

The prediction of human gaze behavior is important for building human-co...

Predicting Visual Attention and Distraction During Visual Search Using Convolutional Neural Networks

Most studies in computational modeling of visual attention encompass tas...

What am I Searching for: Zero-shot Target Identity Inference in Visual Search

Can we infer intentions from a person's actions? As an example problem, ...

Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attention

Predicting human gaze is important in Human-Computer Interaction (HCI). ...

What am I searching for?

Can we infer intentions and goals from a person's actions? As an example...

Benchmarking human visual search computational models in natural scenes: models comparison and reference datasets

Visual search is an essential part of almost any everyday human goal-dir...

Please sign up or login with your details

Forgot password? Click here to reset