Exploring Automated Code Evaluation Systems and Resources for Code Analysis: A Comprehensive Survey

by   Md. Mostafizer Rahman, et al.

The automated code evaluation system (AES) is mainly designed to reliably assess user-submitted code. Due to their extensive range of applications and the accumulation of valuable resources, AESs are becoming increasingly popular. Research on the application of AES and their real-world resource exploration for diverse coding tasks is still lacking. In this study, we conducted a comprehensive survey on AESs and their resources. This survey explores the application areas of AESs, available resources, and resource utilization for coding tasks. AESs are categorized into programming contests, programming learning and education, recruitment, online compilers, and additional modules, depending on their application. We explore the available datasets and other resources of these systems for research, analysis, and coding tasks. Moreover, we provide an overview of machine learning-driven coding tasks, such as bug detection, code review, comprehension, refactoring, search, representation, and repair. These tasks are performed using real-life datasets. In addition, we briefly discuss the Aizu Online Judge platform as a real example of an AES from the perspectives of system design (hardware and software), operation (competition and education), and research. This is due to the scalability of the AOJ platform (programming education, competitions, and practice), open internal features (hardware and software), attention from the research community, open source data (e.g., solution codes and submission documents), and transparency. We also analyze the overall performance of this system and the perceived challenges over the years.


A Survey on Online Judge Systems and Their Applications

Online judges are systems designed for the reliable evaluation of algori...

Automated Generation of Computer Graded Unit Testing-Based Programming Assessments for Education

Automatic assessment of code, in particular to support education, is an ...

Is ChatGPT the Ultimate Programming Assistant – How far is it?

The recent progress in generative AI techniques has significantly influe...

Building Automated Survey Coders via Interactive Machine Learning

Software systems trained via machine learning to automatically classify ...

A Study of Editor Features in a Creative Coding Classroom

Creative coding is a rapidly expanding domain for both artistic expressi...

The Impact of Auto-Refactoring Code Smells on the Resource Utilization of Cloud Software

Cloud-based software-as-a-service (SaaS) have gained popularity due to t...

Chatbots As Fluent Polyglots: Revisiting Breakthrough Code Snippets

The research applies AI-driven code assistants to analyze a selection of...

Please sign up or login with your details

Forgot password? Click here to reset