AI Inference Costs Drop Below $1 per Million Tokens in 2026

The price of a token has fallen below a dollar

The AI industry's most closely watched number in 2026 is the cost of serving a model, not the cost of training one, and that number is falling faster than most budgets expected. Silicon Data's LLM Token Expenditure Index, a usage-weighted measure of what buyers pay, dropped to $0.97 per million tokens on September 1, 2026. That is the lowest reading since the index launched in late 2025, and it is less than half the peak the index reached earlier in the summer.38 Separate tracking put average prices at about $2.04 per million tokens at the end of May, falling to roughly $1.16–$1.18 by early August. That is a 43% drop in about ten weeks.3135

The longer trend is steeper still. GPT-4 launched in March 2023 at $30 per million input tokens. By mid-2026, open-weight models with comparable or better quality cost under $0.50, which works out to roughly a 1,000x decline over three years at a fixed capability level.32 Analysts who follow the market describe the cost of reaching a given capability as falling about 10x a year. Epoch AI's estimates put the median decline nearer 50x a year, with some tasks falling as fast as 900x.38

The headline numbers make it tempting to credit new chips for the drop. A close look at the 2026 coverage suggests something else. The summer's price collapse came mostly from competition and model efficiency. The new hardware arriving now will mainly decide who can keep cutting prices profitably once the price war ends.

What actually drove the summer collapse

Reports agree on what caused this year's sharpest declines, and none of the causes is a new chip. OpenAI cut prices on its GPT-5.6 Luna models by 80% in late July, bringing input tokens to $0.20 per million and output tokens to $1.20.31 Open-weight Chinese models, from DeepSeek and others, pushed the floor down further, with some providers reporting cuts of up to 99% earlier in the year.31 One index analysis singles out Moonshot's Kimi K3 as a model that reset the floor. It also notes that the pressure lands hardest on OpenAI and Anthropic, both widely seen as IPO candidates.38

Model design is the other big factor. Mixture-of-experts architectures, which activate only part of a model's parameters for each token, are credited with roughly a 3–5x cut in compute per token.3336 Quantization, speculative decoding and routing requests to smaller models add further savings. One estimate says these techniques together cut compute per token by 5–20x without losing quality.32

These are mostly software and pricing choices. That matters for the hardware story: Nvidia's next-generation platform was not yet serving customers at scale while prices were falling fastest.

Nvidia's 10x promise and its fine print

Nvidia introduced its Rubin platform at CES on January 5, 2026. The company said it would cut inference token costs by up to 10x compared with Blackwell and train mixture-of-experts models with a quarter as many GPUs.1216 The launch described six co-designed chips: the Vera CPU, Rubin GPU, NVLink 6 switch, ConnectX-9 SuperNIC, BlueField-4 DPU and Spectrum-6 Ethernet switch. Each Rubin GPU delivers 50 petaflops of NVFP4 compute, and partner availability was promised for the second half of 2026.16

The platform has changed since January. At GTC, Nvidia added a seventh chip, the Groq 3 LPX, a low-latency accelerator built for the decode phase of agentic workloads.17 Reports disagree on whether GTC took place in March or April.1317 One outlet even placed the Rubin unveiling at Computex.18 The point the coverage agrees on is that Rubin has become a full-rack system rather than a single GPU. In a July technical blog, Nvidia cited 22 TB/s of HBM4 bandwidth per GPU, about 2.8x Blackwell's 8 TB/s, as the main reason Rubin can serve bandwidth-hungry MoE models more cheaply.17

The caveats matter. The 10x figure is an upper bound measured against Blackwell, on workloads Nvidia chose: agentic AI, advanced reasoning and large-scale MoE inference.1316 One analysis notes that the savings depend on all seven chips working together as one system. In practice, a buyer cannot swap in a cheaper component and still get the advertised cost.17 That makes Rubin's economics inseparable from deeper dependence on Nvidia's full stack.

A FinOps analysis offers a useful reality check. It describes the 10x claim as a silicon-level efficiency figure, not a promise about anyone's bill. It notes that AWS raised some GPU prices by about 15% in early 2026 and that GPU supply remains tight.15 AWS, Google Cloud, Microsoft Azure and Oracle, along with CoreWeave, Lambda, Nebius and Nscale, were named as early Rubin cloud providers for the second half of the year.17 Some commentators forecast that cloud token prices will bottom out around the first quarter of 2027.18 That is speculation, but the timing fits: most of Rubin's effect on prices is still ahead.

Google's Ironwood and the cost of custom chips

Google's answer is Ironwood, its seventh-generation TPU. It became generally available at Cloud Next on April 22, and Google called it its first TPU built for the age of inference.21 Each chip delivers 4.6 petaflops, and a 9,216-chip superpod reaches 42.5 exaflops.24 Google also previewed an eighth generation split into two chips for the first time: TPU 8t, a training chip designed with Broadcom, and TPU 8i, an inference chip designed with MediaTek. Both target TSMC's 2nm process in late 2027.21 Google claims TPU 8i will offer 80% better inference price-performance than Ironwood.24

Ironwood's pricing shows the biggest gap between list price and what large customers actually pay. Anthropic is the anchor customer, with access to up to one million TPUs and more than a gigawatt of capacity in 2026. About 400,000 of those units come as finished racks from Broadcom, worth roughly $10 billion.2122 SemiAnalysis estimates that Anthropic pays about $1.60 per TPU-hour.22 When Google finally published an on-demand rate months after launch, it was $12.00 an hour in Iowa. B200 GPUs on other clouds were listed at $9.36 an hour on-demand.23 Google's own figures put Ironwood's total cost of ownership about 44% below an Nvidia GB200 server in a full pod. Even so, no independent inference benchmark had been published by mid-2026.2226

The takeaway is that custom silicon is cheap for customers who commit at enormous scale and relatively expensive for everyone else. Ironwood is available only through Google Cloud, with no third-party resale.23

Hyperscaler chips spread

Google is not the only one. Microsoft's Maia 200 is an inference-only chip on TSMC 3nm with 216GB of HBM3e and about 10 petaflops of FP4 compute at 750 watts. It is reportedly running GPT-5.2 in production, and a Microsoft executive called it 30% cheaper than any other AI silicon on the market.6 One report says the chip reached volume about six months later than first planned.10 AWS says Trainium offers 30–40% better price-performance than comparable instances.9 It also claims up to 50% savings over Nvidia-based instances for supported workloads.1

One industry analysis estimates custom ASICs grow at a 44.6% compound annual rate, against 16.1% for general-purpose GPUs.1 Another puts their total-cost advantage over GPUs at 40–65%, as hyperscalers guide toward roughly $700 billion or more in capital spending for 2026.2 These are vendor and analyst projections, not audited results. They all point the same way, though: buyers with steady, high-volume inference are moving those workloads onto chips they control.

Why cheaper tokens still mean bigger bills

The point most reports agree on is that cheaper tokens have not made AI cheaper to run. Reasoning models can generate far more internal tokens than they return to the user. One analysis describes workloads where per-token prices fell 10x while token consumption grew 100x.34 Another estimates that inference makes up 80–90% of an AI product's lifetime spending on models.37 This is the Jevons paradox applied to AI: when tokens get cheaper, agents and long-context workflows become affordable, and total consumption rises faster than prices fall.3237

Forecasts diverge on how long the decline can last. One guide expects annual declines to slow to 3–5x through 2027 and then 1.5–2x.34 More aggressive predictions have frontier output tokens falling from about $15 to $3–5 per million by early 2027.18 This analysis leans toward the more cautious view for frontier models, which index trackers note fall in price more slowly than older models that have become commodities.38

The bottom line

In 2026, inference costs fell because of a price war fought with better models and thinner margins, before most next-generation hardware was even running. Rubin, Ironwood, Maia and Trainium will decide what happens after that. Their makers are competing to have the lowest cost per token by the time model prices stop falling, which some forecast for around early 2027. Nvidia is offering a tightly integrated rack. The hyperscalers are offering cheaper capacity, but mostly to customers willing to commit to a single cloud. For most buyers, the key cost in 2026 is the price they negotiate, not the chip's spec sheet.