AI Inference Costs Fall 1,000x as Rubin and TPUs Battle Over Tokens

The price of yesterday's frontier keeps falling

The central fact in AI hardware this year has little to do with peak FLOPS. What matters is that a token from a GPT-4-class model now costs a tiny fraction of what it did at launch. Analyses published through 2026 widely describe a decline of roughly 1,000x over about three years for a fixed level of capability. One widely shared estimate puts GPT-4-equivalent inference at about $20 per million tokens in late 2022 and about $0.40 by early 202612. Another starts from GPT-4's March 2023 launch price of $30 per million input tokens and says comparable output now costs less than $0.50 through open-weight models such as Llama 3.3 70B and DeepSeek V311.

That consensus is looser than it looks. Ask what the starting price was, which model counts as "equivalent," and over what window, and the coverage starts to disagree. The 1,000x number comes mainly from Andreessen Horowitz's "LLMflation" analysis. It tracked GPT-3-level quality falling from $60 per million tokens in late 2021 to about $0.06 by late 2024, or roughly 10x a year16. Stanford HAI's AI Index found a 280-fold drop for GPT-3.5-level quality between late 2022 and October 2024. Epoch AI found declines ranging from 9x to 900x per year depending on the benchmark, with a median of 50x16. One cost tracker hedges its own range at 150x to 1,000x, depending on the model14. One of the 1,000x write-ups also concedes that at the frontier-quality level, the measured drop is closer to 60x11.

Read together, the numbers say this. Cost at a fixed capability level is collapsing by at least one order of magnitude a year, and the exact multiple depends on the benchmark. "1,000x" is a reasonable shorthand for that trend. It is not a measured property of any one product.

Why it compounds

Analysts mostly agree that no single breakthrough explains the decline. They describe several gains multiplying together: each GPU generation brings two to three times more inference throughput per dollar, serving software adds batching and memory-management tricks, mixture-of-experts (MoE) models activate only part of their parameters for each token, and quantization lowers precision12. Some estimates credit MoE alone with a 3-5x cut in compute per token15. Isolated algorithmic progress has been estimated at about 3x a year16.

This matters for the hardware story because it means silicon is only one lever. Chipmakers' marketing tends to suggest otherwise, but model design and serving software have done at least as much of the work.

Nvidia sells the next 10x

Nvidia still pitches its roadmap in cost-per-token terms. At CES in January 2026, Jensen Huang introduced the Vera Rubin NVL72 rack. Nvidia described it as six co-designed chips: the Vera CPU, the Rubin GPU, an NVLink 6 switch, a ConnectX-9 SuperNIC, a BlueField-4 DPU and a Spectrum-6 Ethernet switch21. The company claimed each Rubin GPU delivers 50 petaflops of NVFP4 inference, five times Blackwell GB200, and up to 10x lower cost per token for MoE inference21. It also said training MoE models would take a quarter as many GPUs, with volume production planned for the second half of 20262129. Later reporting from GTC described the platform as seven chips in full production22.

The 10x figure deserves more skepticism than it got. One technical breakdown notes that Nvidia measured it on the Kimi-K2-Thinking MoE model, with 32K-token inputs and 8K-token outputs, comparing a Vera Rubin NVL72 against a GB200 NVL7223. The same analysis estimates that dense-model inference will improve closer to 2-3x over Blackwell, and that short-context dense workloads will fall well short of 10x23. Even the baseline isn't consistent across outlets. Most compare Rubin with Blackwell2126, but at least one European outlet framed the 10x as a gain over Hopper. It then suggested 5-7x was a more realistic figure for production workloads24.

The pricing math still favors buyers with the right workloads. Rubin NVL72 racks are estimated at $3.5 million to $4 million, about a 25% premium over Blackwell racks23. That means a buyer pays more per rack and, if Nvidia's claims hold, much less per token. My read is that the 10x claim is credible for the workloads that dominate frontier serving in 2026: large, sparse, long-context reasoning models. Enterprises running smaller dense models should plan around something closer to 2-3x.

Custom silicon is now a real alternative

The biggest structural change is that custom chips have moved beyond internal projects at the hyperscalers. Google's seventh-generation TPU, Ironwood, is built mainly for inference. It delivers 4,614 FP8 teraflops per chip with 192GB of HBM3E, and scales to 9,216-chip pods producing 42.5 exaflops5. Sources disagree on its timeline. Some say it became generally available in late 2025 after its April 2025 announcement15. Others tie general availability to Google Cloud Next in April 2026, where Google also previewed an eighth generation split into separate training (TPU 8t) and inference (TPU 8i) chips27.

Customers matter more than specs here. Anthropic's October 2025 agreement gives it access to up to one million TPUs and more than a gigawatt of capacity in 202635. Meta reportedly signed a multiyear, multibillion-dollar lease for TPU capacity in February 2026, with an option to buy Ironwood hardware outright from 20278. Google is also reported to run more than 75% of Gemini computation on Ironwood10. Amazon's Trainium 3, Microsoft's Maia 200 and Meta's MTIA chips are all aimed at the same serving workloads14.

Ironwood's public pricing shows how custom silicon competes. Google lists on-demand Ironwood at $12 an hour in Iowa, falling to $5.40 on a three-year commitment9. A SemiAnalysis estimate puts Anthropic's negotiated rate at about $1.60 per TPU-hour9. If that figure is accurate, the biggest customers pay a fraction of the list price. Those discounts are why frontier labs are spreading their workloads across vendors.

Some reporting overstates the case. One outlet went further than others and said Anthropic already runs more than a million Ironwood chips4, while most describe the commitment as access to "up to" a million37. One review notes that a year after launch, Ironwood still has no independent inference benchmark. Google's strongest claims depend on its own serving stack3. These chips are also captive: Ironwood, Trainium and Maia can only be rented from the companies that make them, and Ironwood depends on JAX rather than the CUDA-based serving tools most teams use15.

Nvidia's share is under pressure, not collapsing

None of this has dented Nvidia's revenue yet. Estimates of its share of the AI accelerator market cluster around 73-81%58. One analysis puts its quarterly data center revenue at $89 billion, against $6.7 billion for AMD's data center segment8. Projections that Nvidia's share of inference could fall to 20-30% by 2028 circulate widely, but they remain projections1. The better reading is that custom silicon is taking the biggest, most predictable inference workloads, where a single owner controls the whole stack. Nvidia keeps the merchant market and anyone who needs portability. Both camps also depend on TSMC, which is a bottleneck neither side controls10.

Cheaper tokens, bigger bills

Most of the coverage agrees on one paradox: lower unit costs have not reduced total spending. Enterprise generative-AI spending grew from $2.3 billion in 2023 to $37 billion in 2025, according to Menlo Ventures, even as token prices fell sharply18. Gartner estimates that agentic queries use 5-30 times more tokens than simple prompts16. Google's monthly token throughput reportedly reached 3.2 quadrillion, up sevenfold year over year16. One analysis notes that H100 rental rates have held steady or risen even as cost per token fell, because demand keeps absorbing the new capacity12. A FinOps review warns that Rubin will not automatically lower cloud bills, and points to tight GPU supply and an AWS price increase of about 15% on some GPU instances in early 202626.

The effect on the chip business follows from this. A 1,000x fall in cost per token has not shrunk the market. It has widened the set of workloads worth running, from agent loops to real-time voice13. As spending shifts toward inference, the companies best positioned are the ones that can deliver the lowest cost per token at scale. For now that means Nvidia's rack-scale systems in the open market and the hyperscalers' own chips inside their clouds, with cost per token, more than FLOPS, deciding who wins.