Active Bayesian Assessment for Black-Box Classifiers

by   Disi Ji, et al.
University of California, Irvine

Recent advances in machine learning have led to increased deployment of black-box classifiers across a wide variety of applications. In many such situations there is a crucial need to assess the performance of these pre-trained models, for instance to ensure sufficient predictive accuracy, or that class probabilities are well-calibrated. Furthermore, since labeled data may be scarce or costly to collect, it is desirable for such assessment be performed in an efficient manner. In this paper, we introduce a Bayesian approach for model assessment that satisfies these desiderata. We develop inference strategies to quantify uncertainty for common assessment metrics (accuracy, misclassification cost, expected calibration error), and propose a framework for active assessment using this uncertainty to guide efficient selection of instances for labeling. We illustrate the benefits of our approach in experiments assessing the performance of modern neural classifiers (e.g., ResNet and BERT) on several standard image and text classification datasets.


Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models

A wide variety of natural language tasks are currently being addressed w...

Ensembling Uncertainty Measures to Improve Safety of Black-Box Classifiers

Machine Learning (ML) algorithms that perform classification may predict...

Probabilistic supervised learning

Predictive modelling and supervised learning are central to modern data ...

Toward Optimal Probabilistic Active Learning Using a Bayesian Approach

Gathering labeled data to train well-performing machine learning models ...

Differential Assessment of Black-Box AI Agents

Much of the research on learning symbolic models of AI agents focuses on...

Bayesian Quantification with Black-Box Estimators

Understanding how different classes are distributed in an unlabeled data...

Uncertainty in Fairness Assessment: Maintaining Stable Conclusions Despite Fluctuations

Several recent works encourage the use of a Bayesian framework when asse...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset