DeepAI AI Chat
Log In Sign Up

Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?

by   Chengwei Qin, et al.
Northeastern University

Prompt tuning (PT) which only tunes the embeddings of an additional sequence of tokens per task, keeping the pre-trained language model (PLM) frozen, has shown remarkable performance in few-shot learning. Despite this, PT has been shown to rely heavily on good initialization of the prompt embeddings. In this work, we study meta prompt tuning (MPT) to systematically explore how meta-learning can help improve (if it can) cross-task generalization in PT through learning to initialize the prompt embeddings from other relevant tasks. We empirically analyze a representative set of meta learning algorithms in a wide range of adaptation settings with different source/target task configurations on a large set of few-shot tasks. With extensive experiments and analysis, we demonstrate the effectiveness of MPT. We find the improvement to be significant particularly on classification tasks. For other kinds of tasks such as question answering, we observe that while MPT can outperform PT in most cases, it does not always outperform multi-task learning. We further provide an in-depth analysis from the perspective of task similarity.


page 28

page 29


Boosting Natural Language Generation from Instructions with Meta-Learning

Recent work has shown that language models (LMs) trained with multi-task...

Similarity of Classification Tasks

Recent advances in meta-learning has led to remarkable performances on s...

Adaptable Text Matching via Meta-Weight Regulator

Neural text matching models have been used in a range of applications su...

CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models

Despite the recent advances showing that a model pre-trained on large-sc...

Effective Structured Prompting by Meta-Learning and Representative Verbalizer

Prompt tuning for pre-trained masked language models (MLM) has shown pro...

MetaPrompting: Learning to Learn Better Prompts

Prompting method is regarded as one of the crucial progress for few-shot...

Structured Prompt Tuning

We propose structured prompt tuning, a simple and effective method to im...