Online Active Model Selection for Pre-trained Classifiers

by   Mohammad Reza Karimi, et al.

Given k pre-trained classifiers and a stream of unlabeled data examples, how can we actively decide when to query a label so that we can distinguish the best model from the rest while making a small number of queries? Answering this question has a profound impact on a range of practical scenarios. In this work, we design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and stochastic streams. We establish several theoretical guarantees for our algorithm and extensively demonstrate its effectiveness in our experimental studies.


page 1

page 2

page 3

page 4


Cost-Effective Online Contextual Model Selection

How can we collect the most useful labels to learn a model selection pol...

Selective Sampling and Imitation Learning via Online Regression

We consider the problem of Imitation Learning (IL) by actively querying ...

Cost-Effective Training of Deep CNNs with Active Model Adaptation

Deep convolutional neural networks have achieved great success in variou...

Message Passing Adaptive Resonance Theory for Online Active Semi-supervised Learning

Active learning is widely used to reduce labeling effort and training ti...

Active and Continuous Exploration with Deep Neural Networks and Expected Model Output Changes

The demands on visual recognition systems do not end with the complexity...

Stream-based active learning with linear models

The proliferation of automated data collection schemes and the advances ...

Model Selection for Production System via Automated Online Experiments

A challenge that machine learning practitioners in the industry face is ...

Please sign up or login with your details

Forgot password? Click here to reset