Randomized ReLU Activation for Uncertainty Estimation of Deep Neural Networks

07/15/2021
by   Yufeng Xia, et al.
47

Deep neural networks (DNNs) have successfully learned useful data representations in various tasks, however, assessing the reliability of these representations remains a challenge. Deep Ensemble is widely considered the state-of-the-art method for uncertainty estimation, but it is very expensive to train and test. MC-Dropout is another alternative method, which is less expensive but lacks the diversity of predictions. To get more diverse predictions in less time, we introduce Randomized ReLU Activation (RRA) framework. Under the framework, we propose two strategies, MC-DropReLU and MC-RReLU, to estimate uncertainty. Instead of randomly dropping some neurons of the network as in MC-Dropout, the RRA framework adds randomness to the activation function module, making the outputs diverse. As far as we know, this is the first attempt to add randomness to the activation function module to generate predictive uncertainty. We analyze and compare the output diversity of MC-Dropout and our method from the variance perspective and obtain the relationship between the hyperparameters and output diversity in the two methods. Moreover, our method is simple to implement and does not need to modify the existing model. We experimentally validate the RRA framework on three widely used datasets, CIFAR10, CIFAR100, and TinyImageNet. The experiments demonstrate that our method has competitive performance but is more favorable in training time and memory requirements.

READ FULL TEXT

page 7

page 11

page 12

research
12/15/2020

Masksembles for Uncertainty Estimation

Deep neural networks have amply demonstrated their prowess but estimatin...
research
07/07/2020

Single Shot MC Dropout Approximation

Deep neural networks (DNNs) are known for their high prediction performa...
research
08/18/2023

Robust Uncertainty Quantification using Conformalised Monte Carlo Prediction

Deploying deep learning models in safety-critical applications remains a...
research
10/12/2021

Robust Neural Regression via Uncertainty Learning

Deep neural networks tend to underestimate uncertainty and produce overl...
research
10/08/2020

Prediction intervals for Deep Neural Networks

The aim of this paper is to propose a suitable method for constructing p...
research
10/13/2021

Dropout Prediction Variation Estimation Using Neuron Activation Strength

It is well-known DNNs would generate different prediction results even g...
research
10/07/2019

Evaluating Scalable Uncertainty Estimation Methods for DNN-Based Molecular Property Prediction

Advances in deep neural network (DNN) based molecular property predictio...

Please sign up or login with your details

Forgot password? Click here to reset