Inference Cost Per Token Plunges as Rubin and Ironwood Compete

The number that now runs the AI industry

For most of the generative AI boom, chips were sold on peak FLOPS. By late 2026 the industry is selling something else: cost per million tokens served. Nvidia, Google and the benchmarking firms that check them now pitch hardware mainly on that figure. The coverage points in two directions at once. The cost of producing a token is falling very fast. The total amount spent on AI infrastructure is rising even faster. To understand the economics of AI hardware this year, you have to hold both facts together.

Nvidia's Rubin pitch: one-tenth the cost per token

Nvidia set the terms in January at CES. It launched the Vera Rubin platform there as a six-chip rack-scale system and said it would deliver up to 10x lower inference token cost than Blackwell. It also said Rubin could train mixture-of-experts (MoE) models with a quarter as many GPUs.3 The company said each Rubin GPU delivers 50 petaflops of NVFP4 compute and that partner availability would follow in the second half of 2026.3 Tom's Hardware reported the matching claim of up to 5x greater inference performance. It also covered a new storage tier built on BlueField-4 DPUs, meant to handle key-value cache operations that Nvidia says become the bottleneck once context windows reach millions of tokens.1

By GTC the platform had grown to seven chips in full production.2 Nvidia now lists Groq 3 LPX among the parts that must work together to reach its efficiency numbers.7 By midsummer the pitch was more about power. Nvidia cited a CoreWeave benchmark on DeepSeek-R1 showing 10x more throughput per megawatt than Grace Blackwell NVL72. It said Google Cloud's new A5X instances, built on Vera Rubin NVL72, offer up to 10x lower inference cost per token.10

The move from "cheaper" to "more tokens per megawatt" is telling. Operators that can't get enough electricity now care more about how much they can produce inside a fixed power budget than about chip prices.

Where the 10x claim holds, and where it doesn't

No outlet disputes that Rubin is a big step up. They do disagree about how far the 10x figure applies. One technical breakdown argues the gain depends heavily on MoE routing patterns and long sequence lengths. For dense transformer inference, it estimates the improvement over Blackwell at closer to 2x to 3x.5 The same analysis puts a Vera Rubin NVL72 rack at roughly $3.5 million to $4.0 million, about a 25% premium over Blackwell. Per-hour rental prices should therefore rise even if per-token costs fall.5 A European CIO-focused outlet suggests planning for a 5x to 7x reduction in real production workloads rather than the headline figure.4

My reading: the 10x number is real but conditional. It describes the large sparse MoE models frontier labs are shipping, served with Nvidia's full rack and software stack. That stack-wide dependence also works as lock-in. One report notes the claimed gains rely on all seven chips operating together as one system.7

Software moves the cost more than silicon

The most useful data this year comes from SemiAnalysis's InferenceX benchmarks. They show that the same hardware can produce very different results depending on configuration. Running DeepSeek-V4-Pro at the same interactivity level costs $0.28 per million tokens on a GB200 NVL72 and $0.12 on a GB300 NVL72, a 57% drop in one hardware generation.38 On a single GB300 rack serving DeepSeek R1, turning multi-token prediction on cut the cost from $2.35 to $0.11 per million tokens. That is a 21x swing from one software setting.38

Nvidia makes the same point in its own favor. It says software tuning alone brought GPT-OSS-120B on B200 down to two cents per million tokens, a 5x cut in two months.38 The benchmarks also show Nvidia doesn't win every test. On GLM-5 in FP8 with multi-token prediction, AMD's Instinct MI355X came in about 27% cheaper than a B200 in the same configuration.38

The implication is uncomfortable for anyone comparing spec sheets. Comparing chips on cost per token means little unless the model, precision, serving stack and latency target are all held fixed.

Google's Ironwood puts a price on custom silicon

The biggest challenge to Nvidia's cost lead came from Google's seventh-generation TPU, Ironwood. For months after general availability in April, Google published no list price. The only figure in circulation was a SemiAnalysis estimate that Anthropic pays about $1.60 per TPU-hour under a negotiated deal.21 In September, Google's pricing pages finally listed Ironwood at $12.00 per chip-hour on demand in Iowa and $13.20 in London.22 Flex-start capacity costs $6.00, and one- and three-year commitments cost $8.40 and $5.40.24 Those commitment discounts work out to about 30% and 55% off on-demand.30

On hourly price alone, Ironwood isn't cheap. One GPU marketplace listed B200 on demand at $9.36 an hour, below Ironwood's Iowa rate.22 The cost-per-token figures tell a different story. In SemiAnalysis's InferenceX test at 100 tokens per second per user, Ironwood came in at $0.181 per million tokens. B200 came in at $0.222 and B300 at $0.276.29 That makes Ironwood about 18% cheaper than B200 and 34% cheaper than B300 at that operating point.23 At a 20-second median response-time target, the gap narrows to about 8% against B200.23

The caveats matter. B200 pulled ahead in part of the tested range, and Nvidia kept its edge when models used FP4.27 Observers have also asked whether Ironwood's lead holds on long-context prefill or on models other than the Qwen3.5 397B system tested.29 Ironwood is available only on Google Cloud, while B200 runs across multiple clouds and on-premises.25 Google is already promising that TPU 8i will bring an 80% performance-per-dollar gain over Ironwood for low-latency MoE inference.27

So custom silicon is now competitive on cost per token, but within narrow bounds. Its lead is real and modest, it applies only to certain workloads, and buyers have to be willing to stay inside one cloud. SemiAnalysis also reports that Anthropic has become the largest TPU user, with commitments of more than one million units.29 That kind of anchor customer is what makes the investment pay off.

Rental-market figures don't line up

Cost-per-token figures from rental markets vary widely, and they should be read with care. One GPU cloud estimates an H100 running a 70B model in FP8 at high concurrency at about $0.227 per million tokens on demand.37 The same provider's FP16 numbers for Llama 70B on an 8x H100 node come out near $2.30 per million.39 Precision and batch size explain most of that tenfold difference. Its own data shows that at a batch size of one, cost per million tokens on H100 rises above $250.39 The headline numbers depend on utilization assumptions, and real deployments may never reach them.

Prices also differ widely by provider. The same 8x H100 configuration costs about $1.90 per million tokens on a specialist cloud and about $5.46 on AWS.31 Market trackers put the average H100 on-demand rate near $3.61 per GPU-hour, within a range of roughly $1.25 to $14.90.35

Why bills keep rising anyway

The paradox is clearest in usage figures. Google reportedly processed 3.2 quadrillion tokens a month by mid-2026, about seven times more than a year earlier. OpenAI's 2025 inference bill was reported at around $8.4 billion, roughly four times the year before, even as per-token prices fell.35 Cheaper tokens lead people to use far more of them, a pattern known as Jevons' paradox.

The hardware market isn't getting cheaper either. One FinOps analysis notes that AWS raised some GPU prices by about 15% in early 2026. It also notes that tight supply sets what customers pay, regardless of silicon efficiency.8 An October survey of the market found that rents on deployed GPUs rose this year because frontier cluster time and energized power are scarce. At the same time, token prices are in a price war that grows usage faster than spending.20

The capex behind the price war

All of this rests on an unusually large build-out. Trackers put 2026 capex for Amazon, Alphabet, Meta and Microsoft at about $730 billion, up 78% from about $410 billion in 2025.11 Estimates differ depending on who is counted and how. CreditSights puts the top five near $602 billion, J.P. Morgan says $697 billion15, and Goldman Sachs says about $800 billion.19 Goldman expects hyperscaler AI spending to rise more than 50% in 2027 to $1.2 trillion.12

The funding picture is shifting. Epoch AI's analysis of SEC filings suggests hyperscaler cash capex will exceed operating cash flow around the third quarter of 2026. Debt has grown from 9% of capex in fiscal 2024 to 32% on a trailing basis.15 S&P Global Ratings expects all six of the largest hyperscalers to have negative free operating cash flow in 2026 and 2027.17 Goldman estimates the hyperscalers need about $300 billion a year in AI revenue to break even on this cohort of spending.20

The bottom line

Cost per token is now the industry's main metric, but it measures operating efficiency, not whether the business is profitable. Rubin's 10x is probably real for the MoE workloads frontier labs care about most. Ironwood shows custom silicon can beat Nvidia on cost per token by modest margins. Configuration choices can still move costs more than a new chip generation does. None of this has made AI cheaper to run overall. Lower token costs are being spent on more tokens: longer contexts, agent loops and reasoning chains. The open question is whether enough revenue grows from that extra usage to cover the debt now financing the build-out.