Responsible Active Learning via Human-in-the-loop Peer Study

by   Yu-Tong Cao, et al.

Active learning has been proposed to reduce data annotation efforts by only manually labelling representative data samples for training. Meanwhile, recent active learning applications have benefited a lot from cloud computing services with not only sufficient computational resources but also crowdsourcing frameworks that include many humans in the active learning loop. However, previous active learning methods that always require passing large-scale unlabelled data to cloud may potentially raise significant data privacy issues. To mitigate such a risk, we propose a responsible active learning method, namely Peer Study Learning (PSL), to simultaneously preserve data privacy and improve model stability. Specifically, we first introduce a human-in-the-loop teacher-student architecture to isolate unlabelled data from the task learner (teacher) on the cloud-side by maintaining an active learner (student) on the client-side. During training, the task learner instructs the light-weight active learner which then provides feedback on the active sampling criterion. To further enhance the active learner via large-scale unlabelled data, we introduce multiple peer students into the active learner which is trained by a novel learning paradigm, including the In-Class Peer Study on labelled data and the Out-of-Class Peer Study on unlabelled data. Lastly, we devise a discrepancy-based active sampling criterion, Peer Study Feedback, that exploits the variability of peer students to select the most informative data to improve model stability. Extensive experiments demonstrate the superiority of the proposed PSL over a wide range of active learning methods in both standard and sensitive protection settings.


page 1

page 2

page 3

page 4


Distribution Aware Active Learning

Discriminative learning machines often need a large set of labeled sampl...

Multi-class Active Learning: A Hybrid Informative and Representative Criterion Inspired Approach

Labeling each instance in a large dataset is extremely labor- and time- ...

Active Learning for Argument Mining: A Practical Approach

Despite considerable recent progress, the creation of well-balanced and ...

An overview of active learning methods for insurance with fairness appreciation

This paper addresses and solves some challenges in the adoption of machi...

Active Learning with Partial Feedback

In the large-scale multiclass setting, assigning labels often consists o...

Active Learning with Contrastive Pre-training for Facial Expression Recognition

Deep learning has played a significant role in the success of facial exp...

Using Active Learning Methods to Strategically Select Essays for Automated Scoring

Research on automated essay scoring has become increasing important beca...

Please sign up or login with your details

Forgot password? Click here to reset