No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence

by   Chaozheng Wang, et al.

Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.


page 2

page 3

page 4

page 5

page 7

page 9

page 11

page 12


SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models

The pre-training and fine-tuning paradigm has contributed to a number of...

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Large Language Models (LLMs) possess impressive capabilities to generate...

AUG-FedPrompt: Practical Few-shot Federated NLP with Data-augmented Prompts

Transformer-based pre-trained models have become the de-facto solution f...

Generating Query Focused Summaries without Fine-tuning the Transformer-based Pre-trained Models

Fine-tuning the Natural Language Processing (NLP) models for each new da...

Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks

While fine-tuning pre-trained models for downstream classification is th...

Plug-and-Play Document Modules for Pre-trained Models

Large-scale pre-trained models (PTMs) have been widely used in document-...

Prompt-Tuning Decision Transformer with Preference Ranking

Prompt-tuning has emerged as a promising method for adapting pre-trained...

Please sign up or login with your details

Forgot password? Click here to reset