Correct-by-Construction Parsing
Parsers are some of the most important pieces of software that we have. You have likely relied on dozens of parsers to even read this message. Anytime someone writes a piece of high-level code, it must be parsed into a representation that is workable for the machine.
Because parsers are so ubiquituous, the correctness of so many other programs is contingent on a bug-free parser. Despite this fundamental reliance, parsers are often unverified. Buggy parsers have led to many critical security vulnerabilities. Even in otherwise verified software development, parsing is usually unverified (or the final component to become verified).
To address this gap, we develop Lambek, a domain-specific programming language for writing intrinsically correct parsers. That is, if a parser written in Lambek compiles then we guarantee the resulting program is sound with respect to its parsing semantics.
I first explored this idea in Intrinsic Verification of Parsers and Formal Grammar Theory in Dependent Lambek Calculus, and it will be the focus of my dissertation.
Notes and sketches of ongoing work in Lambek may be found below and throughout this site.
30 entries
Definition 1. Representable Grammars representable-grammar
For any string , we can define a representable grammar which matches exactly the string and nothing else. The parse trees for a representable grammar are proofs that the string is exactly equal to :
2 Intrinsic Verification of Parsers and Formal Grammar Theory in Dependent Lambek Calculus pldi-2025-talk
We present Dependent Lambek Calculus, a domain-specific dependent type theory for verified parsing and formal grammar theory. In Dependent Lambek Calculus, linear types are used as a syntax for formal grammars, and parsers can be written as linear terms. The linear typing restriction provides a form of intrinsic verification that a parser yields only valid parse trees for the input string. We demonstrate the expressivity of this system by showing that the combination of inductive linear types and dependency on non-linear data can be used to encode commonly used grammar formalisms such as regular and context-free grammars as well as traces of various types of automata. Using these encodings, we define parsers for regular expressions using deterministic automata, as well as examples of verified parsers of context-free grammars.
We present a denotational semantics of our type theory that interprets the types as a mathematical notion of formal grammars. Based on this denotational semantics, we have made a prototype implementation of Dependent Lambek Calculus using a shallow embedding in the Agda proof assistant. All of our examples parsers have been implemented in this prototype implementation.
3 Intrinsic Verification of Parsers and Formal Grammar Theory in Dependent Lambek Calculus intrinsic-verification-of-parsers
We present Dependent Lambek Calculus (Lambek), a domain-specific dependent type theory for verified parsing and formal grammar theory. In Lambek, linear types are used as a syntax for formal grammars, and parsers can be written as linear terms. The linear typing restriction provides a form of intrinsic verification that a parser yields only valid parse trees for the input string. We demonstrate the expressivity of this system by showing that the combination of inductive linear types and dependency on non-linear data can be used to encode commonly used grammar formalisms such as regular and context-free grammars as well as traces of various types of automata. Using these encodings, we define parsers for regular expressions using deterministic automata, as well as examples of verified parsers of context-free grammars.
We present a denotational semantics of our type theory that interprets the linear types as functions from strings to sets of abstract parse trees and terms as parse transformers. Based on this denotational semantics, we have made a prototype implementation of Lambek using a shallow embedding in the Agda proof assistant. All of our examples parsers have been implemented in this prototype implementation.
Definition 4. A Grammar of Finite Cardinals finite-cardinal-grammar
Define a grammar for each finite cardinal .
Then define the grammar of all finite cardinals as
Definition 5. Finite Unambiguity finite-unambiguity
Define a grammar to be finitely unambiguous if .
There is likely a better name for this.
6 Finite Unambiguity is Not Equivalent to Unambiguity finite-unambiguity-not-unambiguity
For a while I believed finite unambiguity to be equivalent to the other definitions of unambiguity.
Definition 6.1. Finite Unambiguity finite-unambiguity
Define a grammar to be finitely unambiguous if .
There is likely a better name for this.
Definition 6.2. Unambiguity as Subterminality unambiguity-as-subterminality
A grammar is unambiguous if the unique map into the terminal object is a monomorphism. That is, is a subobject of .
Definition 6.3. Unambiguity as Unique Map into Codomain unambiguity-as-unique-map
A grammar is unambiguous if for all grammars and maps we have .
In a category with terminal objects, this is equivalent to unambiguity defined via subterminality.
Definition 6.3.1. Unambiguity as Subterminality unambiguity-as-subterminality
A grammar is unambiguous if the unique map into the terminal object is a monomorphism. That is, is a subobject of .
We may use an analogy from the category of sets, however I had missed the infinite case when translating this idea to grammars.
It is true that if a grammar is unambiguous, then it is finitely unambiguous. However, the converse does not hold unless the grammar has finitely many parse trees for each string. This finiteness condition is a semantic one. If this proof of unambiguity were to be internalized, then it could maybe be captured through the lens of some grammar of finite cardinals.
Let . is finitely unambiguous but not unambiguous. The isomorphism between and amounts to building a bijection between and , which is straightforward.
7 What Connectives Preserve Finiteness? finiteness-preserving-connectives
In order to bridge the gap between finite unambiguity and unambiguity, we can try to restrict to grammars that have finitely many parse trees. Weβd expect this to address the concern semantically, as that fixes the problem when the parses are interpreted in .
We can define when a grammar is finite, and then try to prove that finiteness is preserved on some sane operations on grammars. For instance, , , and each preserve finiteness. I havenβt proven this myself (which would be a useful exercise), but the following is substantiated in any topos with a natural numbers object [johnstone-2002].
7.1 Finite Cardinal Arithmetic in a Topos finite-cardinal-arithmetic-topos
In a topos with a natural numbers object , you can define finite cardinals as objects that arise as the pullback along a morphism of the generic finite cardinal. Write for the cardinal corresponding to .
The above characterization is a little obtuse and does warrant some more explanation. One way to make it more concrete is that and , and that finite cardinals warrant a nice induction principle. If is a property expressible in the internal language such that satisfies , and that whenever satisfies then satisfies ; then every finite cardinal satisfies . That is, forms a -closed subobject of and thus is all of .
My working mental model is in a presheaf topos, where the natural numbers object can be defined explicitly as .
Definition 7.1.1. Constant Presheaf constant-presheaf
The constant presheaf of a set on a category is a functor such that
I often call this the discrete presheaf for , but I donβt know if thatβs standard.
The finite cardinals with respect to can be characterized then as,
These obey nice algebraic properties.
The case of is more problematic. Semantically, for all we may bound the size of the set of parse trees
Precisely knowing this bound isnβt too important, but certainly it exists. We could capture this behavior by just adding an axiom that if and are each finite, then is finite. Although, Iβd rather not add an axiom.
We could directly try to prove that preserves finiteness. The statement would follow from showing that preserves monomorphisms. That is, if and , it suffices to show that is a monomorphism. This would imply finiteness, because you may then apply this for and as . However, you would also need to show that is finite, which isnβt immediately clear.
8 Internal Finiteness internally-finite-grammar
Define a grammar to be finite (or perhaps subfinite) if it is a subobject of grammar of finite cardinals.
Where is defined as follows.
Definition 8.1. A Grammar of Finite Cardinals finite-cardinal-grammar
Define a grammar for each finite cardinal .
Then define the grammar of all finite cardinals as
9 Star Continuity is a Semantic Property star-continuity-semantic-property
Star continuity in Dependent Lambek Calculus can often be a convenient proof technique, but itβs important to remember that this shouldnβt be the first line of defense.
Star continuity holds in the Agda model, but does it hold in the syntactic model? I believe that it does because of the presence of the indexed coproducts. So perhaps it isnβt so sinister after all. It is worth noting that much of the reasoning performed by inducting on the length of a Kleene star isnβt very elegant. If a proof necessitates star continuity, then it doesnβt seem to be aided greatly by the type system.
Definition 10. Unambiguity as Subterminality unambiguity-as-subterminality
A grammar is unambiguous if the unique map into the terminal object is a monomorphism. That is, is a subobject of .
Definition 11. Unambiguity as Unique Map into Codomain unambiguity-as-unique-map
A grammar is unambiguous if for all grammars and maps we have .
In a category with terminal objects, this is equivalent to unambiguity defined via subterminality.
Definition 11.1. Unambiguity as Subterminality unambiguity-as-subterminality
A grammar is unambiguous if the unique map into the terminal object is a monomorphism. That is, is a subobject of .
Definition 12. Unambiguity via the Diagonal Being an Isomorphism unambiguity-via-diagonal
Finite unambiguity does not serve as an adequate definition of unambiguity that is equivalent to unambiguity as subterminality and unambiguity as a unique map. However, the definition attempted via finite unambiguity can be refined to something that is equivalent to these.
A grammar is unambiguous if is an isomorphism. Equivalently, is unambiguous if and are equal.
13 Subobject Classifier in Dependent Lambek Calculus subobject-classifier-for-grammars
Previously in Agda we had constructed equalizers in Lambek using sigma types. Further, equalizers form subobjects, which may be comprehended as maps from a type into a subobject classifier.
I believe in our formalization we want to generalize this idea to a broader class of subobjects, maybe even all of them. That is, we could define a subobject classifying grammar . Then we can internally define any predicate on a grammar via a term .
That is, forall we have , and
The code for this is nearly identical to the definition of equalizers as sigma types, and I have even built a translation of the equalizers code that is defined with this as its foundation. I have not yet tested if either implementation is preferable.
It isnβt yet clear how to best expose this sort of construct syntactically. I suppose you could assume some subobject classifying grammar, but then I donβt know if you can then reap the benefits of the propositions internally. That is, how do you reflect the proposition that two terms are equal in the internal language of propositions rather than the external one?
If nothing else, this code gives me a reusable interface to axiomatize smaller, sandboxed ways in which Iβd like to internalize certain types of propositions (such as βtwo terms are equalβ or βdoes not begin with the character β).
The hope for this code is that it lets me inductively prove the follow last soundness of Kleene star.
Definition 14. Kleene Star in Dependent Lambek Calculus kleene-star
For a grammar , the Kleene star is defined as a least-fixed point,
Definition 15. Star Continuity in Dependent Lambek Calculus star-continuity-in-dependent-lambek
For a grammar , the Kleene star is isomorphic to an indexed coproduct.
That is, we may view the parses of like a linear list comprising parses of concatenated together. Further, for each of these lists we may know the precise length.
When viewing Dependent Lambek Calculus as a model of Kleene algebra, this is precisely the statement that star continuity holds.
Definition 16. First Sets in Dependent Lambek Calculus first-set-in-dependent-lambek
The first set of a grammar may be captured in Lambek via the following proposition:
Or perhaps with ones of the grammars
Definition 17. FollowLast Sets in Dependent Lambek Calculus followlast-set-in-dependent-lambek
The followlast set of a grammar may be captured in Lambek via the following proposition:
Or perhaps with ones of the grammars
Definition 18. Nullability in Dependent Lambek Calculus nullability-in-dependent-lambek
The nullability () of a grammar may be captured in Lambek via the following proposition:
Or perhaps with one of the grammars
19 Intrinsic Verification of Parsers and Formal Grammar Theory in Dependent Lambek Calculus mwpls-2024-talk
We present Dependent Lambek Calculus, a domain-specific dependent type theory for verified parsing and formal grammar theory. In Dependent Lambek Calculus, linear types are used as a syntax for formal grammars, and parsers can be written as linear terms. The linear typing restriction provides a form of intrinsic verification that a parser yields only valid parse trees for the input string. We demonstrate the expressivity of this system by showing that the combination of inductive linear types and dependency on non-linear data can be used to encode commonly used grammar formalisms such as regular and context-free grammars as well as traces of various types of automata. Using these encodings, we define parsers for regular expressions using deterministic automata, as well as examples of verified parsers of context-free grammars.
We present a denotational semantics of our type theory that interprets the types as a mathematical notion of formal grammars. Based on this denotational semantics, we have made a prototype implementation of Dependent Lambek Calculus using a shallow embedding in the Agda proof assistant. All of our examples parsers have been implemented in this prototype implementation.
Definition 20. The Quotient and its Right Adjoint in Day Convolution day-quotient
Let be a small monoidal category. Recall that for presheaves , the Day convolution provides a closed monoidal structure:
which forms an adjunction .
For covariant functors and presheaves , we can define the quotient and its right adjoint, each of which is a presheaf on :
These form the adjunction .
Definition 21. Dependent Tensor of Grammars dependent-tensor
The tensor of grammars,
is simply typed, so cannot see what matched.
Just as we have the generalization from the pair type to the dependent pair , we can define a dependent tensor operation. It is a -like generalization over the ordinary tensor, in which the second grammar depends on a parse tree of the first.
Write for the type of parse trees of , each paired with the string it parses.
For a grammar and a family of grammars , the dependent tensor is:
We may also write it as .
When is a constant family, the dependent tensor reduces to the ordinary tensor .
Dependence runs left to right here, which fits left-to-right parsing. The other handedness, in which the first grammar depends on a parse of the second, is also definable.
Relation to Day Convolution
Just as the ordinary tensor is given by Day convolution, I think there is a similar dependent Day convolution for which this operation is an instance. My best guess is for any presheaf on a monoidal category and a functor from the category of elements of into presheaves on ,
Iβm not positive on this though, and I would hope to also extend this to enrichments that arenβt but I donβt see how that could make sense given that I have quantified over the element in the coend rather than using the tensor of the enriching category.
Definition 22. Formal Grammar formal-grammar
For an alphabet , let be the free monoid of strings over . A formal grammar is a family of types indexed by strings:
This definition views formal grammars directly as indexed families of types over strings (which equivalently form presheaves on the discrete category of strings). This view forms the central notion of the Dependent Lambek Calculus. For a given string , the type represents the type of all valid parse trees for according to the grammar . If the grammar cannot parse , then is the empty type.
Definition 23. The Derivative of a Grammar grammar-derivative
When a grammar quotient is taken with respect to a representable grammar (matching exactly the string ), it coincides with the residual , as established by the Quotient and Residual Coincidence.
This special case is known as the Brzozowski derivative of by the string , often written as . We have the following coincidence:
Definition 24. Later on Grammars grammar-later
Write when is a proper suffix of , that is, with . The later of a grammar is
A parse of over is a parse of over every proper suffix of . In particular is a singleton.
This is later on families for strings under the proper-suffix order. That order is well-founded because it strictly decreases length, so it is a thin direct category. There is at most one map , so the product over maps from the strict past has one factor per proper suffix. Guarded recursion is modelled by presheaves on , the topos of trees, and more generally by sheaves over a well-founded base [1]. Here the later acts on families, which is what grammars are, and there is no clock.
Later is the right adjoint of the proper-suffix derivative. Let be the grammar of non-empty strings. Then the derivative has the right adjoint
so . Splitting into its summands gives the form of that the calculus can define:
The component at says: if the string begins with , then the rest parses as .
The restriction to non-empty is what makes this a later. At the component is . Including it would give a projection , and LΓΆb would then prove every grammar. For the same reason the later is a product over suffixes. A sum such as is empty at , and at it is . The identity step would then give LΓΆb a proof of .
In the Agda this is β· in Grammar/Later/Base.agda, which is defined as the indexed conjunction of βl-string. The mirror image β·r, over proper prefixes, is defined in the same way. See also the bilateral later and the later along an arbitrary well-founded order.
Theorem 25. LΓΆb Induction for Grammars grammar-lob
For every grammar and every term , where is the later on grammars, there is a unique global parse with
Here restricts a global parse to every proper suffix.
The proof is recursion on the length of the string. At , the parses already built at the proper suffixes of form an element of , and turns it into a parse at . Uniqueness is LΓΆb for families over the proper-suffix order. In the Agda, lob in Grammar/Later/Base.agda is this recursion, done by well-founded induction on length.
To prove an entailment this way, apply LΓΆb to . The hypothesis is the induction hypothesis at every proper suffix. It becomes usable once a non-nullable grammar has been consumed. If , then
because a parse of over splits with non-empty, so . This is β·-app-NE in Grammar/Later/Properties.agda. Induction on a Kleene star is the standard use.
Theorem 26. Next on Grammars Is Presheaf Structure grammar-next-presheaf
For presheaves, restricts along the strict past, as in later on presheaves. A grammar is only a family over strings, so a map into the later on grammars is extra data. It sends a parse over to parses over every proper suffix of .
Let , so that . This is the comonad for the suffix order, and presheaves are comonadic over families. The counit law forces the first component of a coalgebra to be the identity. Hence:
- a presheaf on strings under the suffix order is the same as a grammar with a map that satisfies coassociativity. Restricting to and then to must agree with restricting to directly;
- for a proposition-valued grammar (a language), coassociativity is automatic, and exists exactly when the language is closed under suffixes. For example, has one, but does not, since is empty.
LΓΆb does not need on . The fixed-point equation only restricts a global parse , and a global parse can always be restricted. In the Agda, the grammar-level IsCoalgebra record, with only the field next, is in Grammar/Later/Coalgebra.agda on the guarded branch.
On presheaves the strict downset of is represented by , because every proper suffix of is a suffix of . So predecessors simplify later to . On grammars there is no such simplification, since the factors at different suffixes are unrelated.
Definition 27. Grammar Quotients grammar-quotients
For formal grammars and , the quotient of by is the grammar of what is left of once an has been read off the front:
A parse of the quotient consists of an -parses at prefix and a -parse of the full string .
This operation has a right adjoint:
Intuitively, parses if: for all splittings of into a prefix and suffix, if the prefix matches then the suffix must match .
Together, these form the adjunction:
This is an instance of the Day quotient over the discrete monoidal category of strings.
Definition 28. Grammar Residuals grammar-residual
For formal grammars and , the residual (sometimes called the lollipop or linear implication) is the right adjoint to the grammar tensor . It describes strings that, when prefixed by a string matching , will match .
Formally, it is defined as:
Intuitively, a parse of at a string is a function that takes any prefix string and a parse of in , and produces a parse of the full concatenated string in .
Definition 29. Tensor of Grammars grammar-tensor
For formal grammars and , their tensor represents the concatenation of the languages they describe. A parse for at a string consists of a splitting of into a prefix and suffix , along with a parse of in and a parse of in .
Formally, it is defined as:
This operation is exactly the Day convolution of and , where we view formal grammars as presheaves over the monoid of strings.
Definition 30. Types over an Algebraic Theory theory-type
Fix a finitary algebraic theory : sorts , a signature , equations, and a set of generators. Write for the free -model on and for its carrier at sort .
A -type of sort is a family .
For -types and of sort we have the additives, defined pointwise,
along with , , and their indexed versions.
Each operation of gives a multiplicative, defined by Day convolution,
Each lifted operation also has closed structure: a residual in each of its arguments. Day algebras [1] give a more general convolutional definition.
When is the theory of monoids and is an alphabet, -types are formal grammars. Multiplication gives , the unit gives , and we recover Lambek.