Revisiting Saliency Metrics: Farthest-Neighbor Area Under Curve

by   Sen Jia, et al.

Saliency detection has been widely studied because it plays an important role in various vision applications, but it is difficult to evaluate saliency systems because each measure has its own bias. In this paper, we first revisit the problem of applying the widely used saliency metrics on modern Convolutional Neural Networks(CNNs). Our investigation shows the saliency datasets have been built based on different choices of parameters and CNNs are designed to fit a dataset-specific distribution. Secondly, we show that the Shuffled Area Under Curve(S-AUC) metric still suffers from spatial biases. We propose a new saliency metric based on the AUC property, which aims at sampling a more directional negative set for evaluation, denoted as Farthest-Neighbor AUC(FN-AUC). We also propose a strategy to measure the quality of the sampled negative set. Our experiment shows FN-AUC can measure spatial biases, central and peripheral, more effectively than S-AUC without penalizing the fixation locations. Thirdly, we propose a global smoothing function to overcome the problem of few value degrees (output quantization) in computing AUC metrics. Comparing with random noise, our smooth function can create unique values without losing the relative saliency relationship.


page 1

page 3

page 4

page 5

page 6

page 8


Limitations of Pinned AUC for Measuring Unintended Bias

This report examines the Pinned AUC metric introduced and highlights som...

A Modified AUC for Training Convolutional Neural Networks: Taking Confidence into Account

Receiver operating characteristic (ROC) curve is an informative tool in ...

A Distributionally Robust Area Under Curve Maximization Model

Area under ROC curve (AUC) is a widely used performance measure for clas...

What do different evaluation metrics tell us about saliency models?

How best to evaluate a saliency model's ability to predict where humans ...

Metrics for saliency map evaluation of deep learning explanation methods

Due to the black-box nature of deep learning models, there is a recent d...

SalSi: A new seismic attribute for salt dome detection

In this paper, we propose a saliency-based attribute, SalSi, to detect s...

The Area Under the ROC Curve as a Measure of Clustering Quality

The Area Under the the Receiver Operating Characteristics (ROC) Curve, r...

Please sign up or login with your details

Forgot password? Click here to reset