CIFT: Crowd-Informed Fine-Tuning to Improve Machine Learning Ability

by   John P. Lalor, et al.
University of Massachusetts Amherst
Vanderbilt University
University of Massachusetts Medical School

Item Response Theory (IRT) allows for measuring ability of Machine Learning models as compared to a human population. However, it is difficult to create a large dataset to train the ability of deep neural network models (DNNs). We propose Crowd-Informed Fine-Tuning (CIFT) as a new training process, where a pre-trained model is fine-tuned with a specialized supplemental training set obtained via IRT model-fitting on a large set of crowdsourced response patterns. With CIFT we can leverage the specialized set of data obtained through IRT to inform parameter tuning in DNNs. We experiment with two loss functions in CIFT to represent (i) memorization of fine-tuning items and (ii) learning a probability distribution over potential labels that is similar to the crowdsourced distribution over labels to simulate crowd knowledge. Our results show that CIFT improves ability for a state-of-the art DNN model for Recognizing Textual Entailment (RTE) tasks and is generalizable to a large-scale RTE test set.


page 1

page 2

page 3

page 4


Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?

Machine learning practitioners often fine-tune generative pre-trained mo...

An Analysis of Ability in Deep Neural Networks

Deep neural networks (DNNs) have made significant progress in a number o...

Improved Regularization and Robustness for Fine-tuning in Neural Networks

A widely used algorithm for transfer learning is fine-tuning, where a pr...

AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning

With the rapid adoption of machine learning (ML), a number of domains no...

Assessing, testing and estimating the amount of fine-tuning by means of active information

A general framework is introduced to estimate how much external informat...

Building an Evaluation Scale using Item Response Theory

Evaluation of NLP methods requires testing against a previously vetted g...

Fine Tuning with Abnormal Examples

Given the prevalence of crowd sourced labor in creating Natural Language...

Please sign up or login with your details

Forgot password? Click here to reset