Reference. Passport: Improving Automated Formal Verification Using Identifiers
Formally verifying system properties is one of the most effective ways of improving system quality, but its high manual effort requirements often render it prohibitively expensive. Tools that automate formal verification by learning from proof corpora to synthesize proofs have just begun to show their promise. These tools are effective because of the richness of the data the proof corpora contain. This richness comes from the stylistic conventions followed by communities of proof developers, together with the powerful logical systems beneath proof assistants. However, this richness remains underexploited, with most work thus far focusing on architecture rather than on how to make the most of the proof data. This article systematically explores how to most effectively exploit one aspect of that proof data: identifiers. We develop the Passport approach, a method for enriching the predictive Coq model used by an existing proof-synthesis tool with three new encoding mechanisms for identifiers: category vocabulary indexing, subword sequence modeling, and path elaboration. We evaluate our approach’s enrichment effect on three existing base tools: ASTactic, Tac, and Tok. In head-to-head comparisons, Passport automatically proves 29% more theorems than the best-performing of these base tools. Combining the three tools enhanced by the Passport approach automatically proves 38% more theorems than combining the three base tools. Finally, together, these base tools and their enhanced versions prove 45% more theorems than the combined base tools. Overall, our findings suggest that modeling identifiers can play a significant role in improving proof synthesis, leading to higher-quality software.
Cite
Cited by (4)
Cobblestone: A Divide-and-Conquer Approach for Automating Formal Verification kasibatla-2026-cobblestone
QEDCartographer: Automating Formal Verification Using Reward-Free Reinforcement Learning sanchezstern-2025-qedcartographer
Baldur: Whole-Proof Generation and Repair with Large Language Models first-2023-baldur
PRoofster: Automated Formal Verification agrawal-2023-proofster
Cites 78 works (7 here)
With notes (7)
Baldur: Whole-Proof Generation and Repair with Large Language Models first-2023-baldur
PRoofster: Automated Formal Verification agrawal-2023-proofster
Proof repair across type equivalences ringer-2021-proof
REPLica: REPL instrumentation for Coq analysis ringer-2020-replica
QED at Large: A Survey of Engineering of Formally Verified Software ringer-2019-qed
Development of formal proofs of correctness of programs can increase actual and perceived reliability and facilitate better understanding of program specifications and their underlying assumptions. Tools supporting such development have been available for over 40 years, but have only recently seen wide practical use. Projects based on construction of machine-checked formal proofs are now reaching an unprecedented scale, comparable to large software projects, which leads to new challenges in proof development and maintenance. Despite its increasing importance, the field of proof engineering is seldom considered in its own right; related theories, techniques, and tools span many fields and venues. This survey of the literature presents a holistic understanding of proof engineering for program correctness, covering impact in practice, foundations, proof automation, proof organization, and practical proof development.
Verified Software Toolchain appel_vst_2011
Formal verification of a realistic compiler leroy_formal_2009
This paper reports on the development and formal verification (proof of semantic preservation) of CompCert, a compiler from Clight (a large subset of the C programming language) to PowerPC assembly code, using the Coq proof assistant both for programming the compiler and for proving its correctness. Such a verified compiler is useful in the context of critical software and its formal verification: the verification of the compiler guarantees that the safety properties proved on the source code hold for the executable compiled code as well.
External (71)
- Formal mathematics statement curriculum learning (2023)
- VarCLR: variable semantic representation pre-training via contrastive learning (2022)
- Diversity-driven automated formal verification (2022)
- Competition-level code generation with AlphaCode (2022)
- PaLM: Scaling language modeling with pathways (2022)
- Cub Device Scan is Not Deterministic as Described in the Documentation #454 (2022)
- Estimating PaLM’s Training Cost (2022)
- Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers (2022)
- Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm (2022)
- Large Cumulative Sums Appear to be Nondeterministic (#75240) (2022)
- Reproducibility in Deep Learning and Smooth Activations (2022)
- Roosterize: Suggesting Lemma Names for Coq Verification Projects Using Deep Learning (2021)
- Fast and Memory-Efficient Neural Code Completion (2021)
- The Agda Wiki (2021)
- Program synthesis with large language models (2021)
- Evaluating large language models trained on code (2021)
- The Coq Proof Assistant (2021)
- LISA: Language models of ISAbelle proofs (2021)
- Theorem Proving in Lean (2021)
- Proof extraction for logical neural networks (2021)
- Are my deep learning systems fair? An empirical study of fixed-seed training (2021)
- Talia and Joe Chat about Proof Engineering with Copilot (2021)
- Tactic Learning and Proving for the Coq Proof Assistant (2020)
- TacTok: semantics-aware proof synthesis (2020)
- mCoq: mutation analysis for Coq verification projects (2020)
- Big code != big vocabulary: open-vocabulary models for source code (2020)
- Understanding the Difficulty of Training Transformers (2020)
- Deep Generation of Coq Lemma Names Using Elaborated Terms (2020)
- Graph Representations for Higher-Order Logic and Theorem Proving (2020)
- Problems and opportunities in training deep learning software systems (2020)
- Generating correctness proofs with neural networks (2020)
- Experiment Tracking with Weights and Biases (2020)
- GShard: Scaling giant models with conditional computation and automatic sharding (2020)
- Learning to format Coq code using language models (2020)
- Problems and opportunities in training deep learning software systems: an analysis of variance (2020)
- Generative language modeling for automated theorem proving (2020)
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models (2020)
- Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2019)
- Mutation Analysis for Coq (2019)
- Simple High-Level Code for Cryptographic Arithmetic - With Proofs, Without Compromises (2019)
- A survey of methods for explaining black box models (2019)
- HOList: An environment for machine learning of higher-order theorem proving (extended version) (2019)
- GamePad: A learning environment for theorem proving (2019)
- Learning to prove theorems via interacting with proof assistants (2019)
- A regression proof selection tool for coq (2018)
- Hammer for Coq: Automation for Dependent Type Theory (2018)
- Explaining Explanations: An Overview of Interpretability of Machine Learning (2018)
- On the naturalness of proofs (2018)
- piCoq: parallel regression proving for large-scale verification projects (2018)
- Deep Contextualized Word Representations (2018)
- Training Tips for the Transformer Model (2018)
- ICoq: Regression proof selection for large-scale verification projects (2017)
- Example-directed synthesis: a type-theoretic interpretation (2016)
- SerAPI: Machine-friendly data-centric serialization for COQ (2016)
- PHOG: Probabilistic model for code (2016)
- A deep language model for software code (2016)
- Type-and-example-directed program synthesis (2015)
- Neural Machine Translation of Rare Words with Subword Units (2015)
- Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks (2015)
- Verdi: a framework for implementing and formally verifying distributed systems (2015)
- On the localness of software (2014)
- TBCNN: A tree-based convolutional neural network for programming language processing (2014)
- Machine learning: The high interest credit card of technical debt (2014)
- Industrial Use of CompCert on a Safety-Critical Software Product (2014)
- Certified Programming with Dependent Types: A Pragmatic Introduction to the Coq Proof Assistant (2013)
- Complete completion using types and weights (2013)
- Machine Learning in Proof General: Interfacing Interfaces (2013)
- seL4: formal verification of an OS kernel (2009)
- A new algorithm for data compression (1994)
- COLOG-88 (1990)
- The Calculus of Constructions (1986)