Baldur: Whole-Proof Generation and Repair with Large Language Models

by   Emily First, et al.

Formally verifying software properties is a highly desirable but labor-intensive task. Recent work has developed methods to automate formal verification using proof assistants, such as Coq and Isabelle/HOL, e.g., by training a model to predict one proof step at a time, and using that model to search through the space of possible proofs. This paper introduces a new method to automate formal verification: We use large language models, trained on natural language text and code and fine-tuned on proofs, to generate whole proofs for theorems at once, rather than one step at a time. We combine this proof generation model with a fine-tuned repair model to repair generated proofs, further increasing proving power. As its main contributions, this paper demonstrates for the first time that: (1) Whole-proof generation using transformers is possible and is as effective as search-based techniques without requiring costly search. (2) Giving the learned model additional context, such as a prior failed proof attempt and the ensuing error message, results in proof repair and further improves automated proof generation. (3) We establish a new state of the art for fully automated proof synthesis. We reify our method in a prototype, Baldur, and evaluate it on a benchmark of 6,336 Isabelle/HOL theorems and their proofs. In addition to empirically showing the effectiveness of whole-proof generation, repair, and added context, we show that Baldur improves on the state-of-the-art tool, Thor, by automatically generating proofs for an additional 8.7 65.7 research into using large language models for automating formal verification.


page 1

page 2

page 3

page 4


Autoformalization with Large Language Models

Autoformalization is the process of automatically translating from natur...

Passport: Improving Automated Formal Verification Using Identifiers

Formally verifying system properties is one of the most effective ways o...

Proof Repair Across Type Equivalences

We describe a new approach to automatically repairing broken proofs in t...

CoProver: A Recommender System for Proof Construction

Interactive Theorem Provers (ITPs) are an indispensable tool in the arse...

Using Taint Analysis and Reinforcement Learning (TARL) to Repair Autonomous Robot Software

It is important to be able to establish formal performance bounds for au...

Generating Natural Language Proofs with Verifier-Guided Search

Deductive reasoning (drawing conclusions from assumptions) is a challeng...

Automatically Proving Mathematical Theorems with Evolutionary Algorithms and Proof Assistants

Mathematical theorems are human knowledge able to be accumulated in the ...

Please sign up or login with your details

Forgot password? Click here to reset