AI Inference GPU Costs 2026: B200 Rents Climb as TPU Undercuts

Why the cheapest inference hardware is hard to pin down

In late 2026, the best chip for AI inference is not one product. The answer depends on model size, latency targets, software stack and, more and more, which provider you rent from. Three things have changed the picture this year. Blackwell rental prices have jumped. Google's Ironwood TPU now has a public price and its first third-party cost benchmarks. And Nvidia's Vera Rubin platform has started reaching cloud customers with claims of large cuts in token costs.

The main finding across the coverage is that the hourly rental rate is a poor guide to cost. Cost per token is what matters, and it can rank the hardware very differently.

Blackwell rental prices are rising, and trackers disagree on how fast

All the pricing trackers agree that B200 capacity got more expensive during 2026. They disagree on how much. A SemiAnalysis pricing index cited in October coverage put average B200 rental at $8.01 per GPU-hour in late September, up 79% in three months.16 Silicon Data's neocloud index was lower, at $5.93 for B200 and $6.71 for B300 on October 10.13 The Mercatus GPU Index averaged $6.39 across 12 providers in mid-September, with individual listings from $3.74 to $16.14.15

The spread between providers explains most of this. One October roundup called the B200 either a $3.75 card or an $8 card, depending on whether you count the cheapest in-stock listing or a typical on-demand rate.19 At the top end, AWS lists its B200 instance at about $14.24 per GPU-hour.13 Spheron's on-demand B200 rate is $11.12.31 Another analysis of July pricing concluded that the choice of provider often affects a bill more than the choice between a GPU and a TPU.7 That conclusion holds up across every other data point.

H100 pricing is the most confusing part of the record. One recent headline said H100 rentals had halved to $4.20 an hour, but the same piece reports that $4.20 was a median that had risen across 16 providers by October.11 Other trackers report lower figures:

• CloudZero: a September median of $3.25, described as rising14
• Mercatus: an average of $3.8415
• Silicon Data: $2.8213
• Mid-year analysis: rates had climbed about 40% since October 2025 as capacity tightened, reversing a long decline17

The overall direction seems clear even though the exact figures don't match. Hopper capacity is cheap compared with 2024, but it has stopped getting cheaper. Hyperscalers still charge several times the specialist-cloud rate for the same card: about $6.88 on AWS and up to $12.29 on Azure.12

The headline chip is often the wrong choice

Many teams will find the most useful data in a benchmark of 21 GPUs serving a 7B-parameter model on vLLM, a popular open-source serving engine. The B200 had the lowest latency, decoding at 272 tokens per second per user with a 20-millisecond median time to first token. But it cost $14.98 per million output tokens.39 An older RTX A5000 served the same model for $0.64 per million tokens. That is 27 times cheaper than the B300, and still faster than people can read.39 The RTX 5090 came in at $2.24 and the H100 SXM at $7.78.39

Other cost-per-token analyses of larger models, measured differently, reach the opposite ranking. A May 2026 comparison on Llama 3 70B estimated a B200 at $0.42 per million tokens on a discount cloud, cheaper than an H100. Its higher throughput more than made up for the higher hourly price.12 The two results don't actually conflict. Small models with light traffic can't keep a flagship accelerator busy enough to justify its rate. Large, heavily batched models can.

In practice, the guides converge on a rough sizing ladder:

• 7B to 13B models: L40S cards at around $0.96 an hour31
• 13B to 70B models: H100s31
• 70B at FP16 on a single card, or long-context work: the H200's 141GB of memory31
• Very large models or aggressive FP4 quantization: the B200 and B30036

FP4 support is a real dividing line, as the TPU comparison below shows.

For local and desktop inference, the consensus is just as firm. The RTX 5090 is the fastest consumer card.34 A used RTX 3090 with 24GB of memory keeps winning on value.3238 One guide calculated the 3090 at about 132 tokens per second per $1,000 spent, ahead of both the 4090 and the 5090.32

Ironwood gives Nvidia a real cost competitor

Google's seventh-generation TPU, Ironwood, is the most important development in custom silicon this year. It became generally available on April 22.1 For months, though, Google published no list price, which left buyers without the numbers they needed to compare it.1 That has now changed. Google Cloud lists Ironwood at $12.00 per chip-hour on demand in Iowa and $13.20 in London. One-year commitments cost $8.40, three-year commitments $5.40, and flex-start capacity $6.00.410 Anthropic reportedly pays far less: SemiAnalysis estimates about $1.60 per hour under a contract rate no one else can get.4

The first third-party inference benchmark, from SemiAnalysis in September, put Ironwood ahead on cost per token. At 100 tokens per second per user, Ironwood cost about $0.181 per million tokens, against $0.222 for the B200 and $0.276 for the B300. That makes it roughly 18–19% cheaper than the B200 and 34% cheaper than the B300.36 At a 20-second median response time, the lead over the B200 shrank to about 8%.2

The coverage agrees on the numbers but differs on what they mean. Some write-ups call it a formal challenge to Nvidia in inference.3 More careful analyses point to the conditions behind the results:

• At around a 30-second median response time, the B200 pulls ahead.3
• Ironwood loses its advantage when Nvidia GPUs run at FP4 precision, because it has no native FP4 support.3
• At 256 concurrent users, Ironwood took 2.25 times as long as the B300 to return the first token.5 First-token delay is the wait a chatbot user actually notices.

One reasonable reading follows from this. Ironwood looks strong for high-volume batch and throughput work. Nvidia still has the edge for latency-sensitive chat.

List prices complicate things further. At $12 an hour on demand, Ironwood costs more than some B200 marketplace rates. Only Google's committed-use pricing undercuts Blackwell.4 Ironwood also runs only on Google Cloud, and its software means either JAX/XLA or, more recently, a PyTorch route.7 A Meta-built tool called TorchTPU, which lets PyTorch run natively on TPUs, makes it cheaper to try the chip out.5 Even so, for standard vLLM and PyTorch serving, one decision guide still recommends Nvidia's H100 or H200.9

Rubin raises the bar again, starting at the top

In practice, Nvidia's answer is to change the comparison. Vera Rubin launched at CES in January with claims of a 10x cut in inference token costs compared with Blackwell.2125 At GTC in March, Nvidia added a five-rack platform that includes Groq-derived LPU inference racks. It also claimed 10x higher inference throughput per watt at one-tenth the cost per token.22 Each NVL72 rack holds 72 Rubin GPUs with 288GB of HBM4 apiece and is rated at 3.6 exaflops of FP4 inference.24

The headline claims are not entirely consistent. The Verge reported a different figure: training a large mixture-of-experts model at one-seventh the token cost of Blackwell.27 Another account described a 5x inference performance gain at the rack level.26 All of these are Nvidia's own numbers, and none of the coverage includes independent Rubin benchmarks yet.

The hardware is now arriving. Nvidia said in May that production shipments would start this fall.23 In August it put the Rubin system paired with Groq 3 LPX into full production, with Nebius as the first cloud to adopt the LPX racks.28 On October 27, Oracle announced that Vera Rubin NVL72 is generally available on its cloud, with plans for smaller eight-GPU HGX Rubin servers.29

What buyers should do now

Taking the evidence together, Blackwell's price jump looks like a temporary period between generations rather than a lasting state. Demand for B200s has risen while Rubin capacity remains limited to hyperscalers and a few named clouds. That timing matches Nvidia's own description of when Rubin capacity reaches most customers.24 If Rubin delivers even part of its claimed token-cost cuts, Blackwell and Hopper rental rates should come under pressure in 2027. That is an analytical expectation, not something the sources report.

Meanwhile, Ironwood's benchmarks make one point clear: Nvidia is no longer automatically the cheapest option for inference at scale. The practical approach is to measure cost per token on your own model, at the latency you actually need to hit. Then shop across providers before switching chip families, since provider pricing currently moves the bill more than anything else.