Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning

by   Zhen Wang, et al.
The Ohio State University

Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task. Extensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-art methods, including the full finetuning baseline in some cases, despite only tuning 0.035 task-specific parameters.


Multitask Vision-Language Prompt Tuning

Prompt Tuning, conditioning on task-specific learned prompt vectors, has...

Improving Multitask Retrieval by Promoting Task Specialization

In multitask retrieval, a single retriever is trained to retrieve releva...

Zoo-Tuning: Adaptive Transfer from a Zoo of Models

With the development of deep networks on various large-scale datasets, a...

Provable Pathways: Learning Multiple Tasks over Multiple Paths

Constructing useful representations across a large number of tasks is a ...

Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning

Prompt tuning approaches, which learn task-specific soft prompts for a d...

Pruning Pretrained Encoders with a Multitask Objective

The sizes of pretrained language models make them challenging and expens...

Multitask and Transfer Learning for Autotuning Exascale Applications

Multitask learning and transfer learning have proven to be useful in the...

Please sign up or login with your details

Forgot password? Click here to reset