Autoformalization with Large Language Models

by   Yuhuai Wu, et al.

Autoformalization is the process of automatically translating from natural language mathematics to formal specifications and proofs. A successful autoformalization system could advance the fields of formal verification, program synthesis, and artificial intelligence. While the long-term goal of autoformalization seemed elusive for a long time, we show large language models provide new prospects towards this goal. We make the surprising observation that LLMs can correctly translate a significant portion (25.3%) of mathematical competition problems perfectly to formal specifications in Isabelle/HOL. We demonstrate the usefulness of this process by improving a previously introduced neural theorem prover via training on these autoformalized theorems. Our methodology results in a new state-of-the-art result on the MiniF2F theorem proving benchmark, improving the proof rate from 29.6% to 35.2%.


Towards a Mathematics Formalisation Assistant using Large Language Models

Mathematics formalisation is the task of writing mathematics (i.e., defi...

Baldur: Whole-Proof Generation and Repair with Large Language Models

Formally verifying software properties is a highly desirable but labor-i...

Experimental results from applying GPT-4 to an unpublished formal language

Can large language models be used to complete mathematical tasks that ar...

Proof Artifact Co-training for Theorem Proving with Language Models

Labeled data for imitation learning of theorem proving in large librarie...

LeanDojo: Theorem Proving with Retrieval-Augmented Language Models

Large language models (LLMs) have shown promise in proving formal theore...

Towards Autoformalization of Mathematics and Code Correctness: Experiments with Elementary Proofs

The ever-growing complexity of mathematical proofs makes their manual ve...

nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models

A rigorous formalization of desired system requirements is indispensable...

Please sign up or login with your details

Forgot password? Click here to reset