Research into agentic optimization suggests that while AI agents can now navigate complex codebases to improve inference performance, they often spend up to 20% of their time simply trying to understand the relevant code paths within engine files. This shift is prompting a move toward 'decomposing' the compiler, where traditional heuristic-based optimizations are replaced by stable intermediate representations (IRs) and cost models that help agents navigate a smoother search landscape.
From Compilers to Infrastructure
The traditional role of a machine learning compiler—turning a model into a high-performance kernel—is being challenged by agents that can optimize code directly. Rather than being replaced, the compiler is being redefined as a foundation of shared abstractions and reusable tools.
The goal is to provide a stable IR and verification tools, such as those formalized in Lean, to ensure that the solutions generated by agents are correct. This allows agents to focus on exploring optimization strategies without having to reinvent compiler infrastructure at every iteration.
“In this future, we may no longer need a compiler that actually compiles anything, but we will still need compiler infrastructure to understand and verify programs.”
— Make this forward pass as fast as possible
Navigating the Optimization Search Space
Kernel optimization is increasingly viewed as a search and iteration process where tokens and tool calls represent a significant cost. To optimize this search, new frameworks like Together's TKCC use a custom cost model to rank the top-k parallelism configurations using analytical estimates.
By culling options analytically before testing them in a tiered execution setup, these systems identify promising configurations before investing in expensive agentic optimization. This approach helps prevent agents from getting stuck in local optima during the search process.
Smoothing the Landscape by Removing Heuristics
A major shift in compiler design involves removing implicit semantics and heuristics that make the search landscape unpredictable. For example, OpenAI's Triton compiler previously used pattern-matching to automatically insert Fused Multiply-Add (FMA) calls, which could change the numerical results of a kernel.
To make the search landscape smoother and easier for agents to navigate, these automatic passes are being removed in favor of manual insertions in the source code. This ensures that a small change in the code does not trigger a cascade of unpredictable compiler passes that alter the final output.
Trade-offs in Agentic Tooling
While providing agents with more information can improve results, it introduces significant overhead. Research shows that providing human-engineered tools for research and logging can impede throughput, as agents may spend up to 43% of their tool calls simply trying to understand the system.
Chart drawn by HN Digest from figures reported by ianbarber.blog; values are shown as written in the source.
Furthermore, simple prompting to 'make this forward pass as fast as possible' has been shown to outperform some expert-written strategies. This suggests that the most effective path forward involves providing the right tools for verification and analysis rather than trying to guide the agent's search with rigid, pre-defined paths.