Limitations of Assessing Active Learning Performance at Runtime

by   Daniel Kottke, et al.
Universität Kassel

Classification algorithms aim to predict an unknown label (e.g., a quality class) for a new instance (e.g., a product). Therefore, training samples (instances and labels) are used to deduct classification hypotheses. Often, it is relatively easy to capture instances but the acquisition of the corresponding labels remain difficult or expensive. Active learning algorithms select the most beneficial instances to be labeled to reduce cost. In research, this labeling procedure is simulated and therefore a ground truth is available. But during deployment, active learning is a one-shot problem and an evaluation set is not available. Hence, it is not possible to reliably estimate the performance of the classification system during learning and it is difficult to decide when the system fulfills the quality requirements (stopping criteria). In this article, we formalize the task and review existing strategies to assess the performance of an actively trained classifier during training. Furthermore, we identified three major challenges: 1) to derive a performance distribution, 2) to preserve representativeness of the labeled subset, and 3) to correct against sampling bias induced by an intelligent selection strategy. In a qualitative analysis, we evaluate different existing approaches and show that none of them reliably estimates active learning performance stating a major challenge for future research for such systems. All plots and experiments are provided in a Jupyter notebook that is available for download.


page 1

page 2

page 3

page 4


Active Learning: Problem Settings and Recent Developments

In supervised learning, acquiring labeled training data for a predictive...

Probabilistic Active Learning for Active Class Selection

In machine learning, active class selection (ACS) algorithms aim to acti...

Incremental Active Opinion Learning Over a Stream of Opinionated Documents

Applications that learn from opinionated documents, like tweets or produ...

CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output

It is often infeasible or impossible to obtain ground truth labels for m...

Active Learning and Approximate Model Calibration for Automated Visual Inspection in Manufacturing

Quality control is a crucial activity performed by manufacturing enterpr...

Augmented Memory Networks for Streaming-Based Active One-Shot Learning

One of the major challenges in training deep architectures for predictiv...

Data augmentation on-the-fly and active learning in data stream classification

There is an emerging need for predictive models to be trained on-the-fly...

Please sign up or login with your details

Forgot password? Click here to reset