Confidence estimation of classification based on the distribution of the neural network output layer

by   Abdel Aziz Taha, et al.

One of the most common problems preventing the application of prediction models in the real world is lack of generalization: The accuracy of models, measured in the benchmark does repeat itself on future data, e.g. in the settings of real business. There is relatively little methods exist that estimate the confidence of prediction models. In this paper, we propose novel methods that, given a neural network classification model, estimate uncertainty of particular predictions generated by this model. Furthermore, we propose a method that, given a model and a confidence level, calculates a threshold that separates prediction generated by this model into two subsets, one of them meets the given confidence level. In contrast to other methods, the proposed methods do not require any changes on existing neural networks, because they simply build on the output logit layer of a common neural network. In particular, the methods infer the confidence of a particular prediction based on the distribution of the logit values corresponding to this prediction. The proposed methods constitute a tool that is recommended for filtering predictions in the process of knowledge extraction, e.g. based on web scrapping, where predictions subsets are identified that maximize the precision on cost of the recall, which is less important due to the availability of data. The method has been tested on different tasks including relation extraction, named entity recognition and image classification to show the significant increase of accuracy achieved.


Calibrating Structured Output Predictors for Natural Language Processing

We address the problem of calibrating prediction confidence for output e...

Confidence-Nets: A Step Towards better Prediction Intervals for regression Neural Networks on small datasets

The recent decade has seen an enormous rise in the popularity of deep le...

Confidence Ranking for CTR Prediction

Model evolution and constant availability of data are two common phenome...

Learning Confidence for Transformer-based Neural Machine Translation

Confidence estimation aims to quantify the confidence of the model predi...

Bayesian Layer Graph Convolutioanl Network for Hyperspetral Image Classification

In recent years, research on hyperspectral image (HSI) classification ha...

Distance-based Confidence Score for Neural Network Classifiers

The reliable measurement of confidence in classifiers' predictions is ve...

Please sign up or login with your details

Forgot password? Click here to reset