Reference. HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement
Formal methods play a crucial role in ensuring the reliability of critical systems through rigorous mathematical verification. However, their adoption remains limited due to the labor-intensive nature of manual proof construction. Recent advances in large language models (LLMs) have opened new opportunities for automated theorem proving. Two main paradigms have emerged: stepwise tactic-based generation and whole-proof synthesis. While both approaches have complementary strengths, existing work largely treats them in isolation. In this work, we propose HybridProver, a unified framework that integrates whole-proof synthesis and tactic-based generation through proof sketches as an intermediate representation. This design enables the reuse of partially correct proof structures while effectively combining high-level planning with fine-grained reasoning. We implement HybridProver in Isabelle/HOL and post-train two 7B-scale LLMs on our optimized Isabelle datasets. Experiments on the miniF2F Isabelle benchmark achieved a 73.8% success rate and improved upon the previous state of the art (61.9%), demonstrating that lightweight models, when combined with our approach, can effectively generate Isabelle/HOL proofs without relying on very large LLMs. Ablation studies further analyze the impact of dataset quality, training configurations, and sampling strategies on proof generation.
Cite
Cites 72 works (2 here)
With notes (2)
Baldur: Whole-Proof Generation and Repair with Large Language Models first-2023-baldur
Proof Repair Infrastructure for Supervised Models: Building a Large Proof Repair Dataset reichel-2023-proof
We report on our efforts building a new, large proof-repair dataset and benchmark suite for the Coq proof assistant. The dataset is made up of Git commits from open-source projects with old and new versions of definitions and proofs aligned across commits. Building this dataset has been a significant undertaking, highlighting a number of challenges and gaps in existing infrastructure. We discuss these challenges and gaps, and we provide recommendations for how the proof assistant community can address them. Our hope is to make it easier to build datasets and benchmark suites so that machine-learning tools for proofs will move to target the tasks that matter most and do so equitably across proof assistants.
External (70)
- FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning (2026)
- A Minimalist Proof Language for Neural Theorem Proving over Isabelle/HOL (2026)
- DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (2025)
- Tacco: A Framework for Ensuring the Security of Real-World TEEs via Formal Verification (2025)
- MPS-Prover: Advancing Stepwise Theorem Proving by Multi-Perspective Search and Data Curation (2025)
- APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning (2025)
- DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition (2025)
- Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learning (2025)
- Diverse Inference and Verification for Advanced Reasoning (2025)
- BFS-Prover: Scalable Best-First Tree Search for LLM-based Automatic Theorem Proving (2025)
- Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis (2025)
- Aristotle: IMO-level automated theorem proving (2025)
- Seed-Prover: Deep and broad reasoning for automated theorem proving (2025)
- Goedel-Prover: A frontier model for open-source automated theorem proving (2025)
- Isabelle/HOL: A Proof Assistant for Higher-Order Logic (Tutorial) (2025)
- The Isabelle/Isar Reference Manual (2025)
- Correct and Complete Type Checking and Certified Erasure for Coq, in Coq (2024)
- ProveriT: A Parameterized, Composable, and Verified Model of TEE Protection Profile (2024)
- SubgoalXL: Subgoal-based Expert Learning for Theorem Proving (2024)
- DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search (2024)
- TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts (2024)
- FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving (2024)
- A Survey on Large Language Models for Code Generation (2024)
- Enchanting Program Specification Synthesis by Large Language Models using Static Analysis and Program Verification (2024)
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024)
- LEGO-Prover: Neural Theorem Proving with Growing Libraries (2023)
- LeanDojo: Theorem Proving with Retrieval-Augmented Language Models (2023)
- NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models (2023)
- Magnushammer: A Transformer-based Approach to Premise Selection (2023)
- nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models (2023)
- ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics (2023)
- DT-Solver: Automated Theorem Proving with Dynamic-Tree Sampling Guided by Proof-level Value Function (2023)
- Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs (2022)
- Formal Specifications from Natural Language (2022)
- Autoformalization with Large Language Models (2022)
- HyperTree Proof Search for Neural Theorem Proving (2022)
- Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers (2022)
- CICM'22 System Entries (2022)
- MiniF2F: a cross-system benchmark for formal Olympiad-level mathematics (2021)
- LoRA: Low-Rank Adaptation of Large Language Models (2021)
- Proof Artifact Co-training for Theorem Proving with Language Models (2021)
- The Lean 4 Theorem Prover and Programming Language (2021)
- Lisa: Language models of isabelle proofs (2021)
- The isabelle system manual (2021)
- Evaluating large language models trained on code (2021)
- Generative Language Modeling for Automated Theorem Proving (2020)
- Code2Inv: A Deep Learning Framework for Program Verification (2020)
- TRL: Transformers Reinforcement Learning (2020)
- Verifying concurrent, crash-safe systems with Perennial (2019)
- BesFS: A POSIX Filesystem for Enclaves with a Mechanized Safety Proof (2018)
- Certified concurrent abstraction layers (2018)
- First Experiments with Neural Translation of Informal to Formal Mathematics (2018)
- Learning Loop Invariants for Program Verification (2018)
- Refinement-based specification and security analysis of separation kernels (2017)
- Cogent: Verifying High-Assurance File System Implementations (2016)
- Hammering towards QED (2016)
- CompCert - A Formally Verified Optimizing Compiler (2016)
- CertiKOS: An extensible architecture for building certified concurrent OS kernels (2016)
- Push-Button verification of file systems via crash refinement (2016)
- Formal API Specification of the PikeOS Separation Kernel (2015)
- Deep Specifications and Certified Abstraction Layers (2015)
- 40 years of formal methods (2014)
- Three years of experience with Sledgehammer, a Practical Link Between Automatic and Interactive Theorem Provers (2012)
- CVC4 (2011)
- seL4: formal verification of an OS kernel (2009)
- Z3: An Efficient SMT Solver (2008)
- The design and implementation of VAMPIRE (2002)
- Isabelle/HOL: A Proof Assistant for Higher-Order Logic (2002)
- SPASS & FLOTTER version 0.42 (1996)
- Proving theorems recursively