Uncertainty Baselines: Benchmarks for Uncertainty Robustness in Deep Learning

06/07/2021
by   Zachary Nado, et al.
0

High-quality estimates of uncertainty and robustness are crucial for numerous real-world applications, especially for deep learning which underlies many deployed ML systems. The ability to compare techniques for improving these estimates is therefore very important for research and practice alike. Yet, competitive comparisons of methods are often lacking due to a range of reasons, including: compute availability for extensive tuning, incorporation of sufficiently many baselines, and concrete documentation for reproducibility. In this paper we introduce Uncertainty Baselines: high-quality implementations of standard and state-of-the-art deep learning methods on a variety of tasks. As of this writing, the collection spans 19 methods across 9 tasks, each with at least 5 metrics. Each baseline is a self-contained experiment pipeline with easily reusable and extendable components. Our goal is to provide immediate starting points for experimentation with new methods or applications. Additionally we provide model checkpoints, experiment outputs as Python notebooks, and leaderboards for comparing results. Code available at https://github.com/google/uncertainty-baselines.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
07/14/2022

BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen Neural Networks

High-quality calibrated uncertainty estimates are crucial for numerous r...
research
12/22/2019

A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

Evaluation of Bayesian deep learning (BDL) methods is challenging. We of...
research
12/23/2022

Benchmark for Uncertainty Robustness in Self-Supervised Learning

Self-Supervised Learning (SSL) is crucial for real-world applications, e...
research
12/09/2021

The Peril of Popular Deep Learning Uncertainty Estimation Methods

Uncertainty estimation (UE) techniques – such as the Gaussian process (G...
research
09/17/2019

Learn to Estimate Labels Uncertainty for Quality Assurance

Deep Learning sets the state-of-the-art in many challenging tasks showin...
research
11/23/2018

On the Importance of Strong Baselines in Bayesian Deep Learning

Like all sub-fields of machine learning, Bayesian Deep Learning is drive...
research
07/02/2020

Uncertainty Prediction for Deep Sequential Regression Using Meta Models

Generating high quality uncertainty estimates for sequential regression,...

Please sign up or login with your details

Forgot password? Click here to reset