Assessing Project-Level Fine-Tuning of ML4SE Models

by   Egor Bogomolov, et al.

Machine Learning for Software Engineering (ML4SE) is an actively growing research area that focuses on methods that help programmers in their work. In order to apply the developed methods in practice, they need to achieve reasonable quality in order to help rather than distract developers. While the development of new approaches to code representation and data collection improves the overall quality of the models, it does not take into account the information that we can get from the project at hand. In this work, we investigate how the model's quality can be improved if we target a specific project. We develop a framework to assess quality improvements that models can get after fine-tuning for the method name prediction task on a particular project. We evaluate three models of different complexity and compare their quality in three settings: trained on a large dataset of Java projects, further fine-tuned on the data from a particular project, and trained from scratch on this data. We show that per-project fine-tuning can greatly improve the models' quality as they capture the project's domain and naming conventions. We open-source the tool we used for data collection, as well as the code to run the experiments:


Learning code summarization from a small and local dataset

Foundation models (e.g., CodeBERT, GraphCodeBERT, CodeT5) work well for ...

Data Readiness Levels

Application of models to data is fraught. Data-generating collaborators ...

Platypus: Quick, Cheap, and Powerful Refinement of LLMs

We present Platypus, a family of fine-tuned and merged Large Language Mo...

Visualizing a Moving Target: A Design Study on Task Parallel Programs in the Presence of Evolving Data and Concerns

Common pitfalls in visualization projects include lack of data availabil...

Towards Modernising Data Collection and Archive for the Tor Network

CollecTor is developed by Tor Project's Metrics Team for the purpose of ...

Homogeneous Feature Transfer and Heterogeneous Location Fine-tuning for Cross-City Property Appraisal Framework

Most existing real estate appraisal methods focus on building accuracy a...

Software solutions for form-based collection of data and the semantic enrichment of form data

Data collection is an important part of many citizen science projects as...

Please sign up or login with your details

Forgot password? Click here to reset