Automated Feedback Generation for Competition-Level Code

by   Jialu Zhang, et al.

Competitive programming has become a popular way for programmers to test their skills. Large-scale online programming contests attract millions of experienced programmers to compete against each other. Competition-level programming problems are challenging in nature, and participants often fail to solve the problem on their first attempt. Some online platforms for competitive programming allow programmers to practice on competition-level problems as well, and the standard feedback for an incorrect practice submission is the first test case that the submission fails. Often, the failed test case does not provide programmers with enough information to resolve the errors in their code, and they abandon the problem after several more unsuccessful attempts. We present Clef, the first data-driven tool that can generate feedback on competition-level code automatically by repairing programmers' incorrect submissions. The key development is that Clef can learn how to generate repairs for incorrect submissions by examining the repairs that other programmers made to their own submissions over time. Since the differences between an incorrect program and a correct program for the same task may be significant, we introduce a new data structure, merge trees, to capture the changes between submissions. Merge trees are versatile: they can encode both large algorithm-level redesigns and small statement-level alterations. Clef applies the patterns it learns from a database of submissions to generate repairs for new submissions outside the database. We evaluated Clef on six real-world problems from Codeforces, the world's largest platform for competitive programming. Clef achieves 42.1 submissions. Even when given incorrect submissions from programmers who never found the solution to a problem on their own, Clef repairs the users' programs 34.1


page 3

page 4


Data-Driven Feedback Generation for Introductory Programming Exercises

This paper introduces the "Search, Align, and Repair" data-driven progra...

Automated Feedback Generation for Introductory Programming Assignments

We present a new method for automatically providing feedback for introdu...

Competition-Level Code Generation with AlphaCode

Programming is a powerful and ubiquitous problem-solving tool. Developin...

TraceDiff: Debugging Unexpected Code Behavior Using Trace Divergences

Recent advances in program synthesis offer means to automatically debug ...

Automatic Classification of Error Types in Solutions to Programming Assignments at Online Learning Platform

Online programming courses are becoming more and more popular, but they ...

TipsC: Tips and Corrections for programming MOOCs

With the widespread adoption of MOOCs in academic institutions, it has b...

A Survey on Online Judge Systems and Their Applications

Online judges are systems designed for the reliable evaluation of algori...

Please sign up or login with your details

Forgot password? Click here to reset