In the current era of web development, many architectural decisions are often driven by 'vibes'—subjective preferences for developer experience (DX) or marketing claims about speed. However, as development shifts toward an 'agentic' model where AI agents generate code rather than humans typing it by hand, the constraints of DX become less relevant. Nolan Lawson explored this shift by benchmarking how different JavaScript frameworks affect the efficiency and output quality of AI agents tasked with building a non-trivial Pokémon application.
To conduct the experiment, Lawson instructed an AI agent (Opus 5.5) to build a content-heavy application using ten different frameworks or meta-frameworks, including Next.js, Nuxt, SvelteKit, and Angular. The agent was provided with a specification and allowed to use official tools like Model Context Protocol (MCP) skills where available. The results showed a clear correlation between framework complexity and the 'cost' of generation. Simpler architectures like Hono+htmx and Astro were the most efficient, with Hono+htmx costing a median of $2.67 per run. In contrast, more complex frameworks like SvelteKit ($4.28) and Angular ($4.97) were significantly more expensive in terms of both US dollars and wall-clock time, with Angular taking a median of 17.2 minutes to complete.
The benchmark results showed a clear correlation between framework complexity and the cost of generation, with simpler architectures proving more efficient.
Beyond the cost of generation, the benchmark also measured the final application's performance using Lighthouse metrics. The data revealed a performance gap between Multi-Page Application (MPA) leaning frameworks and Single-Page Application (SPA) heavy frameworks. Astro and Hono+htmx achieved the best results, with Astro transferring 0 KiB of client-side JavaScript and Hono+htmx transferring only 18 KiB. Conversely, heavy frameworks like Next.js (142 KiB) and Angular (93 KiB) resulted in higher JavaScript byte counts and higher Total Blocking Time (TBT). Interestingly, 'Vanilla' JavaScript underperformed compared to several frameworks, suggesting that while simplicity is key for agent efficiency, some level of framework abstraction remains beneficial for the agent's success.
While these results provide a clear snapshot of agent behavior, the experiment has specific limitations. The Pokémon app used as a 'holotype' is primarily content-based and lacks the high interactivity of complex enterprise tools like Gmail or Figma. Because the benchmark focuses on a small-scale application, the performance and cost trade-offs might shift significantly when agents are tasked with building highly dynamic, state-heavy systems. Furthermore, the stochastic nature of LLMs means that individual runs can vary wildly, making these results a median representation of agent behavior rather than a guaranteed outcome for every project.
The Pokémon app used as a 'holotype' is primarily content-based and lacks the high interactivity of complex enterprise tools like Gmail or Figma.