DeepAI AI Chat
Log In Sign Up

Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation

by   Jannik Kossen, et al.

We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach, whereas previous methods have focused on Monte Carlo estimates. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWING, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks. We further theoretically analyze ASEs' errors.


page 1

page 2

page 3

page 4


Active Testing: Sample-Efficient Model Evaluation

We introduce active testing: a new framework for sample-efficient model ...

CEREAL: Few-Sample Clustering Evaluation

Evaluating clustering quality with reliable evaluation metrics like norm...

Active Deep Learning Guided by Efficient Gaussian Process Surrogates

The success of active learning relies on the exploration of the underlyi...

Deep Active Learning with a Neural Architecture Search

We consider active learning of deep neural networks. Most active learnin...

Active Learning for Transition State Calculation

The transition state (TS) calculation is a grand challenge for computati...

A generalized framework for active learning reliability: survey and benchmark

Active learning methods have recently surged in the literature due to th...