Quality-Aware Multimodal Saliency Detection via Deep Reinforcement Learning

by   Xiao Wang, et al.

Incorporating various modes of information into the machine learning procedure is becoming a new trend. And data from various source can provide more information than single one no matter they are heterogeneous or homogeneous. Existing deep learning based algorithms usually directly concatenate features from each domain to represent the input data. Seldom of them take the quality of data into consideration which is a key issue in related multimodal problems. In this paper, we propose an efficient quality-aware deep neural network to model the weight of data from each domain using deep reinforcement learning (DRL). Specifically, we take the weighting of each domain as a decision-making problem and teach an agent learn to interact with the environment. The agent can tune the weight of each domain through discrete action selection and obtain a positive reward if the saliency results are improved. The target of the agent is to achieve maximum rewards after finished its sequential action selection. We validate the proposed algorithms on multimodal saliency detection in a coarse-to-fine way. The coarse saliency maps are generated from an encoder-decoder framework which is trained with content loss and adversarial loss. The final results can be obtained via adaptive weighting of maps from each domain. Experiments conducted on two kinds of salient object detection benchmarks validated the effectiveness of our proposed quality-aware deep neural network.


Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query

The excellent performance of Transformer in supervised learning has led ...

Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?

Deep Reinforcement Learning (DRL) connects the classic Reinforcement Lea...

Efficient Halftoning via Deep Reinforcement Learning

Halftoning aims to reproduce a continuous-tone image with pixels whose i...

Benchmarking Perturbation-based Saliency Maps for Explaining Deep Reinforcement Learning Agents

Recent years saw a plethora of work on explaining complex intelligent ag...

Free-Lunch Saliency via Attention in Atari Agents

We propose a new approach to visualize saliency maps for deep neural net...

Explain Your Move: Understanding Agent Actions Using Focused Feature Saliency

As deep reinforcement learning (RL) is applied to more tasks, there is a...

Content Masked Loss: Human-Like Brush Stroke Planning in a Reinforcement Learning Painting Agent

The objective of most Reinforcement Learning painting agents is to minim...

Please sign up or login with your details

Forgot password? Click here to reset