Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language

by   Paul Denny, et al.

GitHub Copilot is an artificial intelligence model for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about the impact it will have on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.


"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers

Recent developments in deep learning have resulted in code-generation mo...

Computing Education in the Era of Generative AI

The computing education community has a rich history of pedagogical inno...

Language Models Can Teach Themselves to Program Better

This work shows how one can use large-scale language models (LMs) to syn...

Promptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code Generators

With their remarkable ability to generate code, large language models (L...

A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes

We propose a model to automatically describe changes introduced in the s...

Computational Thinking in Patch

With the future likely to see even more pervasive computation, computati...

Natural Language Inference over Interaction Space: ICLR 2018 Reproducibility Report

We have tried to reproduce the results of the paper "Natural Language In...

Please sign up or login with your details

Forgot password? Click here to reset