Anthropic and Theorem have demonstrated that large language models (LLMs) can now produce formal correctness proofs for real-world software at a scale previously considered impractical. By using Claude to prove security properties at the assembly code level for over 100 open-source libraries, the team showed that AI can bridge the gap between complex software and the rigorous mathematical models required for formal verification.

A conceptual illustration representing the scalability of formal software verification using large language models.
Anthropic's research demonstrates how LLMs can scale the generation of formal correctness proofs for production software.www.anthropic.com

This image illustrates the core premise of the article: using LLMs to bridge the gap between complex software and rigorous mathematical models.

Scaling Proofs with LLMs

Historically, formal verification was reserved for mission-critical systems like avionics because it required immense human effort; for example, it took 20 person-years to formally verify the seL4 operating system kernel. Anthropic's research shows that LLMs can now automate much of this process, producing proofs for RISC-V assembly code to ensure programs exit normally without memory-safety errors.

This shift allows for the verification of production-grade libraries that were previously too complex to tackle. By using AI to generate proofs against a general model of the execution environment, developers can move toward a workflow where proven security guarantees are integrated into the development lifecycle.

A block chart comparing the proof structures for libcups, freerdp, and libzmq, highlighting identical and edited sections.
Comparison of proof structures for various production-grade libraries, showing the consistency of AI-generated proofs.www.anthropic.com

The chart provides a concrete example of the production-grade libraries that can now be verified using AI-generated proofs.

A line chart showing the share of 36 verification tasks completed by token budget for different AI models, including Opus 5.5.
Comparison of token efficiency across different models for completing formal verification tasks.www.anthropic.com

This chart quantifies the efficiency of different models in completing verification tasks, supporting the discussion on scaling proofs.

“Formal verification makes forall claims, testing makes exists claims--any mistake in any of the hard-to-verify components might lead to the downfall of the proofs.”

— Formal Verification is Not Verifiable

The Trusted Computing Base

A core limitation of formal verification is that it relies on a 'trusted computing base'—the components that must be assumed correct for the proof to hold. This includes the underlying machine semantics, the operating system's process semantics, and the provers themselves.

If any of these hard-to-verify components contain a bug, the resulting proof may be invalid. To mitigate this, researchers are exploring 'binary verification,' which avoids modeling the semantics of high-level programming languages and instead models the simpler semantics of the machine.

Properties vs. Absolute Correctness

Experts argue that the term 'formally verified' is often used loosely and should be replaced with 'formally verified properties.' Because software is inherently complex, it is often impossible to create a uniform notion of correctness for every layer of a system.

While an engine or database might be mathematically well-defined, user-facing components are often driven by aesthetic and subjective demands. The goal is to move toward a stack of verifiable components where specific, first-order properties are proven at the binary level, even while acknowledging that multi-threaded and distributed systems remain significant challenges for the coming years.

A diagram showing 'kiki' shapes for engines and databases and 'bouba' shapes for user interfaces and seats.
A conceptual diagram illustrating the distinction between mathematically verifiable components and subjective user-facing elements.alperenkeles.com

This figure supports the argument that 'formally verified' should be nuanced to 'formally verified properties' due to the complexity of user-facing components.