A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks

by   Angelos Filos, et al.

Evaluation of Bayesian deep learning (BDL) methods is challenging. We often seek to evaluate the methods' robustness and scalability, assessing whether new tools give `better' uncertainty estimates than old ones. These evaluations are paramount for practitioners when choosing BDL tools on-top of which they build their applications. Current popular evaluations of BDL methods, such as the UCI experiments, are lacking: Methods that excel with these experiments often fail when used in application such as medical or automotive, suggesting a pertinent need for new benchmarks in the field. We propose a new BDL benchmark with a diverse set of tasks, inspired by a real-world medical imaging application on diabetic retinopathy diagnosis. Visual inputs (512x512 RGB images of retinas) are considered, where model uncertainty is used for medical pre-screening—i.e. to refer patients to an expert when model diagnosis is uncertain. Methods are then ranked according to metrics derived from expert-domain to reflect real-world use of model uncertainty in automated diagnosis. We develop multiple tasks that fall under this application, including out-of-distribution detection and robustness to distribution shift. We then perform a systematic comparison of well-tuned BDL techniques on the various tasks. From our comparison we conclude that some current techniques which solve benchmarks such as UCI `overfit' their uncertainty to the dataset—when evaluated on our benchmark these underperform in comparison to simpler baselines. The code for the benchmark, its baselines, and a simple API for evaluating new BDL tools are made available at https://github.com/oatml/bdl-benchmarks.


page 3

page 4


A Benchmark of Medical Out of Distribution Detection

There is a rise in the use of deep learning for automated medical diagno...

Uncertainty Baselines: Benchmarks for Uncertainty Robustness in Deep Learning

High-quality estimates of uncertainty and robustness are crucial for num...

Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision

While Deep Neural Networks (DNNs) have become the go-to approach in comp...

How Reliable is Your Regression Model's Uncertainty Under Real-World Distribution Shifts?

Many important computer vision applications are naturally formulated as ...

Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks

Bayesian deep learning seeks to equip deep neural networks with the abil...

An Expert System to Diagnose Spinal Disorders

Objective: Until now, traditional invasive approaches have been the only...

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Modern machine learning methods including deep learning have achieved gr...

Please sign up or login with your details

Forgot password? Click here to reset