AI Inference Costs: Jassy Says 10x Too High, Bets on Trainium

The claim

Amazon CEO Andy Jassy says AI is still too expensive to run. He estimates that inference, the computing used every time a model answers a prompt or carries out a task, costs about ten times more than it should if AI is to spread across ordinary business software. He made the case in an Andreessen Horowitz interview published Friday, October 23, calling inference "an order of magnitude too expensive" for the broad use he expects.22 His proposed fix is Amazon's own silicon. On cutting inference costs, he said the company's chips business is "a big piece of that."21

The timing matters. Amazon reports third-quarter results on October 29, and prediction-market traders on Kalshi put a 96% chance on executives mentioning Trainium during the call.21 The interview reads as an early version of that pitch. It recasts Amazon's custom-chip program as a fix for an economic problem holding back the whole industry, rather than just a way for AWS to buy fewer GPUs.

The barbell and the untapped middle

Jassy describes AI demand as a barbell. At one end, AI labs consume what he called "gobs and gobs of compute." At the other, enterprises use AI for narrow jobs like customer service and fraud detection.21 In his view, the biggest opportunity is the middle: the production software companies already run, most of which has not been rebuilt around AI.27 Korean outlet DigitalToday reported him saying inference costs have fallen a lot over several years but still sit roughly tenfold too high. Companies, he said, will only rework existing applications if they expect the benefits to outweigh the cost.27

Accounts of the interview agree on this point. Benzinga, 24/7 Wall St., and DigitalToday all present cost, not model capability, as the main barrier to enterprise adoption.212527 That reading looks right. A model that is good enough but too expensive to call millions of times a day will stay in a pilot.

The ceiling is also a supply problem. Jassy said Amazon plans about $220 billion in capital spending this year and still cannot meet demand. Big capacity blocks for 2027 are largely sold, and "a good chunk" of 2028 is too.2224 Investors also circulated a clip of him saying he doesn't expect the shortage to end in 2027.25

The Graviton playbook, rerun against Nvidia

Jassy compares the situation to an earlier one. Years ago, he said, Intel dominated cloud server processors, and Amazon could not get it to prioritize price-performance, so Amazon built Graviton. "We saw that same movie in the AI space," he said.21 He did not name Nvidia, and he called the dominant supplier an important partner, but the coverage makes clear who he meant.2123

The numbers behind the pitch are large. On Amazon's first-quarter call in April, Jassy said the custom silicon business (Trainium, Graviton, and Nitro) had passed a $20 billion annual run rate and was growing at triple-digit rates. He said it would be about $50 billion if Amazon counted its own internal purchases the way a merchant chip vendor would.1 He also disclosed more than $225 billion in Trainium revenue commitments, tied to multi-gigawatt deals with Anthropic and OpenAI.1 Those deals are for up to five gigawatts and about two gigawatts respectively, and the OpenAI deal starts ramping in 2027.1 One later compilation, citing the July earnings call, put the chips run rate above $25 billion.8

The reports differ on what the price-performance figure measures. The Friday coverage quotes Jassy saying Trainium beats comparable processors by roughly 30% to 40%.21 His April comments were more specific. He said Trainium2 is about 30% better on price-performance than comparable GPUs, and Trainium3 is another 30% to 40% better than Trainium2.6 Read together, the advantage over GPUs could be larger than the Friday figure suggests, if each generation's gain holds up against the GPUs available at the same time. These are Amazon's own figures, and no independent benchmark has confirmed them.

One detail weakens the idea that Trainium is only a training chip. Amazon says Trainium2 handles most inference on Bedrock, its managed model service.8 Bedrock has more than 125,000 customers, according to the company.9 Amazon, in other words, already runs this inference strategy at scale.

Nvidia is promising the same tenfold cut

There is a complication in Jassy's argument. The tenfold gap he describes is the same improvement Nvidia says its next platform will deliver. At CES in January, Nvidia introduced Vera Rubin NVL72 and promised up to ten times lower cost per token than Blackwell for mixture-of-experts inference, with systems arriving in the second half of 2026.31 Nvidia's product page frames the comparison against GB200 NVL72 on a reasoning-model workload.37 CoreWeave, which brought up the first validated Rubin rack on June 1, made a similar fleet-level claim. As of October, though, no cloud provider had published an hourly price for Rubin.33

AWS is among the clouds deploying Rubin.36 That makes Amazon's position look less like a contest with Nvidia and more like a hedge. Amazon offers Nvidia's hardware to customers who want it, and steers price-sensitive, high-volume inference toward its own chips. Trainium4 is even slated to support Nvidia's NVLink Fusion interconnect, so mixed clusters are possible.58

The careful reading is that Nvidia's tenfold figure is a best-case number for specific workloads. One FinOps analysis warns that customer bills may still rise in the near term, because GPU supply is tight and cheaper tokens tend to increase consumption.38 Jassy's "order of magnitude" is about what customers actually pay, not what silicon can do on a benchmark. Closing that gap depends as much on supply, margins, and competition as on chip design.

Google's TPU sets a measurable benchmark

If Amazon wants to show that custom silicon lowers inference costs, Google has published comparable data first. In September, SemiAnalysis released what it called the first third-party inference benchmark of Google's TPUv7 Ironwood. At 100 tokens per second per user, Ironwood cost about $0.181 per million tokens, against $0.222 on Nvidia's B200 and $0.276 on the B300.20 Coverage differs on rounding, with Ironwood described as either 18% or 19% cheaper than the B200, but the direction is the same everywhere.1415

The same benchmark shows the limits. Around a 30-second median response time, the B200 still came out ahead on cost.20 Nvidia also kept an edge when running FP4 precision, which Ironwood does not support natively.14 Another analysis found Ironwood took 2.25 times as long as the B300 to return the first token at 256 concurrent users. That points toward TPUs for batch work and Nvidia for latency-sensitive chatbots.13 Pricing is also less clear-cut than the headline numbers suggest. Google lists Ironwood at $12 per chip-hour on demand in Iowa, while SemiAnalysis estimates that Anthropic pays about $1.60 under a negotiated deal.12

The Google comparison matters to Amazon for two reasons. First, it shows that a hyperscaler chip can beat Nvidia on cost per token in independent tests, at least for some workloads, which supports Jassy's general thesis. Second, Trainium has no equivalent public benchmark in the coverage reviewed here. Amazon's claims are internally consistent and backed by huge customer commitments, but they remain Amazon's numbers. Some reports say Trainium3 UltraServers match Blackwell on raw compute at about half the total cost of ownership.5 Those figures come from vendor and trade-press framing, not audited results.

The case is strong, but the money is concentrated

On balance, Jassy's argument holds up. Cost is the main barrier to enterprise AI. Custom silicon is one of the few levers a cloud provider fully controls. And Amazon's chip commitments are too large to dismiss as a side project.

The risks are financial, not technical. Trailing free cash flow has turned to a $7.6 billion outflow as AI spending climbs.24 Much of the booked capacity belongs to a few AI labs whose funding and model economics could change, and Jassy did not say how concentrated the 2028 bookings are.24 The Anthropic dependence also cuts both ways. SemiAnalysis reports that Anthropic has become Google's biggest TPU customer.18 Amazon's anchor Trainium customer, in other words, also buys from Google.

The October 29 call will show whether Amazon adds concrete figures to the argument, such as backlog, Trainium's share of Bedrock traffic, or updated capex.25 Until then, Jassy has set the terms of the comparison. The contest among Nvidia's Rubin, Google's TPUs, and Amazon's Trainium will be decided by who delivers the cheapest token at scale.