The AI cost of JavaScript frameworks and benchmarks

The web development community has long relied on intuition rather than empirical evidence when choosing frameworks and tools. In the era of AI coding agents, this reliance on vibes has persisted, with developers frequently rewriting frontends because a certain framework is supposedly better for agents. Developer Nolan Lawson decided to put these claims to the test by running a comprehensive benchmark.

• Designed a non-trivial Pokémon app using PokéAPI to test various JavaScript frameworks
• Instructed AI agents (Opus 5.5) in sandboxed environments to build the application
• Evaluated results based on API token spend, development time, lines of code, and Lighthouse scores
• Ran five iterations per framework to account for the stochastic nature of language models

Experimental Setup and Methodology

To level the playing field, each framework utilized its official Model Context Protocol (MCP) or specific skills. The architecture was not overly specified, giving Opus 5.5 high effort and freedom to use the latest versions. The resulting applications were tested against a robust conformance suite to ensure accuracy and prevent agent cheating.

Given the probabilistic nature of LLMs, variance is inevitable. Some agents wasted tokens reading migration guides or failed to pick up MCP files entirely. However, by running multiple iterations for each framework, consistent patterns regarding cost, code quality, and maintainability began to emerge.

Cost and Efficiency Findings

The results revealed distinct differences in API costs and operational efficiency across frameworks. SvelteKit, Gracile+Lit, and Angular proved significantly more expensive to build. For SvelteKit, this was likely due to agents repeatedly reading migration guides for a new version, while Angular's higher cost tracked closely with its larger total code size.

Interestingly, not all agents made use of available MCPs or skill files. For instance, TanStack Start discovered its skill files in only two out of five runs, and Nuxt's MCP was never utilized. This mirrors real-world AI coding scenarios where developers rarely enforce strict tool usage.

What This Means for Developers

This benchmark demonstrates that framework selection in the age of AI agents carries real economic and performance costs. Moving away from vibe-based decisions toward evidence-based benchmarks helps developers make better architectural choices when building modern web applications.