Practical Active Learning with Model Selection for Small Data

by   Maryam Pardakhti, et al.

Active learning is of great interest for many practical applications, especially in industry and the physical sciences, where there is a strong need to minimize the number of costly experiments necessary to train predictive models. However, there remain significant challenges for the adoption of active learning methods in many practical applications. One important challenge is that many methods assume a fixed model, where model hyperparameters are chosen a priori. In practice, it is rarely true that a good model will be known in advance. Existing methods for active learning with model selection typically depend on a medium-sized labeling budget. In this work, we focus on the case of having a very small labeling budget, on the order of a few dozen data points, and develop a simple and fast method for practical active learning with model selection. Our method is based on an underlying pool-based active learner for binary classification using support vector classification with a radial basis function kernel. First we show empirically that our method is able to find hyperparameters that lead to the best performance compared to an oracle model on less separable, difficult to classify datasets, and reasonable performance on datasets that are more separable and easier to classify. Then, we demonstrate that it is possible to refine our model selection method using a weighted approach to trade-off between achieving optimal performance on datasets that are easy to classify, versus datasets that are difficult to classify, which can be tuned based on prior domain knowledge about the dataset.


page 1

page 2

page 3

page 4


How useful is Active Learning for Image-based Plant Phenotyping?

Deep learning models have been successfully deployed for a diverse array...

Rebuilding Trust in Active Learning with Actionable Metrics

Active Learning (AL) is an active domain of research, but is seldom used...

On the Robustness of Active Learning

Active Learning is concerned with the question of how to identify the mo...

Active Model Selection

Classical learning assumes the learner is given a labeled data sample, f...

Dominant Set-based Active Learning for Text Classification and its Application to Online Social Media

Recent advances in natural language processing (NLP) in online social me...

Deep Active Learning with Adaptive Acquisition

Model selection is treated as a standard performance boosting step in ma...

Active Few-Shot Classification: a New Paradigm for Data-Scarce Learning Settings

We consider a novel formulation of the problem of Active Few-Shot Classi...

Please sign up or login with your details

Forgot password? Click here to reset