Anthropic and Theorem have demonstrated that large language models (LLMs) can now produce formal correctness proofs for real-world software at a scale previously considered impractical. By using Claude to prove security properties at the assembly code level for over 100 open-source libraries, the team showed that AI can bridge the gap between complex software and the rigorous mathematical models required for formal verification.

This image illustrates the core premise of the article: using LLMs to bridge the gap between complex software and rigorous mathematical models.
Scaling Proofs with LLMs
Historically, formal verification was reserved for mission-critical systems like avionics because it required immense human effort; for example, it took 20 person-years to formally verify the seL4 operating system kernel. Anthropic's research shows that LLMs can now automate much of this process, producing proofs for RISC-V assembly code to ensure programs exit normally without memory-safety errors.
This shift allows for the verification of production-grade libraries that were previously too complex to tackle. By using AI to generate proofs against a general model of the execution environment, developers can move toward a workflow where proven security guarantees are integrated into the development lifecycle.

The chart provides a concrete example of the production-grade libraries that can now be verified using AI-generated proofs.

This chart quantifies the efficiency of different models in completing verification tasks, supporting the discussion on scaling proofs.
“Formal verification makes forall claims, testing makes exists claims--any mistake in any of the hard-to-verify components might lead to the downfall of the proofs.”
— Formal Verification is Not Verifiable
The Trusted Computing Base
A core limitation of formal verification is that it relies on a 'trusted computing base'—the components that must be assumed correct for the proof to hold. This includes the underlying machine semantics, the operating system's process semantics, and the provers themselves.
If any of these hard-to-verify components contain a bug, the resulting proof may be invalid. To mitigate this, researchers are exploring 'binary verification,' which avoids modeling the semantics of high-level programming languages and instead models the simpler semantics of the machine.
Properties vs. Absolute Correctness
Experts argue that the term 'formally verified' is often used loosely and should be replaced with 'formally verified properties.' Because software is inherently complex, it is often impossible to create a uniform notion of correctness for every layer of a system.
While an engine or database might be mathematically well-defined, user-facing components are often driven by aesthetic and subjective demands. The goal is to move toward a stack of verifiable components where specific, first-order properties are proven at the binary level, even while acknowledging that multi-threaded and distributed systems remain significant challenges for the coming years.
This figure supports the argument that 'formally verified' should be nuanced to 'formally verified properties' due to the complexity of user-facing components.