Identification of Negative Transfers in Multitask Learning Using Surrogate Models

by   Dongyue Li, et al.

Multitask learning is widely used in practice to train a low-resource target task by augmenting it with multiple related source tasks. Yet, naively combining all the source tasks with a target task does not always improve the prediction performance for the target task due to negative transfers. Thus, a critical problem in multitask learning is identifying subsets of source tasks that would benefit the target task. This problem is computationally challenging since the number of subsets grows exponentially with the number of source tasks; efficient heuristics for subset selection does not always capture the relationship between task subsets and multitask learning performances. In this paper, we introduce an efficient procedure to address this problem via surrogate modeling. In surrogate modeling, we sample (random) subsets of source tasks and precompute their multitask learning performances; Then, we approximate the precomputed performances with a linear regression model that can also be used to predict the multitask performance of unseen task subsets. We show theoretically and empirically that fitting this model only requires sampling linearly many subsets in the number of source tasks. The fitted model provides a relevance score between each source task and the target task; We use the relevance scores to perform subset selection for multitask learning by thresholding. Through extensive experiments, we show that our approach predicts negative transfers from multiple source tasks to target tasks much more accurately than existing task affinity measures. Additionally, we demonstrate that for five weak supervision datasets, our approach consistently improves upon existing optimization methods for multi-task learning.


page 1

page 2

page 3

page 4


Exploring Multitask Learning for Low-Resource AbstractiveSummarization

This paper explores the effect of using multitask learning for abstracti...

Boosting Multitask Learning on Graphs through Higher-Order Task Affinities

Predicting node labels on a given graph is a widely studied problem with...

Data-Efficient Finetuning Using Cross-Task Nearest Neighbors

Language models trained on massive prompted multitask datasets like T0 (...

Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back

Deep multitask learning boosts performance by sharing learned structure ...

TaskWeb: Selecting Better Source Tasks for Multi-task NLP

Recent work in NLP has shown promising results in training models on lar...

Learning Functions to Study the Benefit of Multitask Learning

We study and quantify the generalization patterns of multitask learning ...

Consistent Multitask Learning with Nonlinear Output Relations

Key to multitask learning is exploiting relationships between different ...

Please sign up or login with your details

Forgot password? Click here to reset