Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learning

by   Mai Xu, et al.

When watching omnidirectional images (ODIs), subjects can access different viewports by moving their heads. Therefore, it is necessary to predict subjects' head fixations on ODIs. Inspired by generative adversarial imitation learning (GAIL), this paper proposes a novel approach to predict saliency of head fixations on ODIs, named SalGAIL. First, we establish a dataset for attention on ODIs (AOI). In contrast to traditional datasets, our AOI dataset is large-scale, which contains the head fixations of 30 subjects viewing 600 ODIs. Next, we mine our AOI dataset and determine three findings: (1) The consistency of head fixations are consistent among subjects, and it grows alongside the increased subject number; (2) The head fixations exist with a front center bias (FCB); and (3) The magnitude of head movement is similar across subjects. According to these findings, our SalGAIL approach applies deep reinforcement learning (DRL) to predict the head fixations of one subject, in which GAIL learns the reward of DRL, rather than the traditional human-designed reward. Then, multi-stream DRL is developed to yield the head fixations of different subjects, and the saliency map of an ODI is generated via convoluting predicted head fixations. Finally, experiments validate the effectiveness of our approach in predicting saliency maps of ODIs, significantly better than 10 state-of-the-art approaches.


page 3

page 5

page 10

page 13


Modeling Attention in Panoramic Video: A Deep Reinforcement Learning Approach

Panoramic video provides immersive and interactive experience by enablin...

Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

We introduce a large-scale dataset of human actions and eye movements wh...

Adversarial Exploration Strategy for Self-Supervised Imitation Learning

We present an adversarial exploration strategy, a simple yet effective i...

Rethink AI-based Power Grid Control: Diving Into Algorithm Design

Recently, deep reinforcement learning (DRL)-based approach has shown pro...

Multi-Scale Estimation for Omni-Directional Saliency Maps Using Learnable Equator Bias

Omni-directional images have been used in wide range of applications. Fo...

Individualizing Head-Related Transfer Functions for Binaural Acoustic Applications

A Head Related Transfer Function (HRTF) characterizes how a human ear re...

Development of an automatic 3D human head scanning-printing system

Three-dimensional (3D) technologies have been developing rapidly recent ...

Please sign up or login with your details

Forgot password? Click here to reset