llm-d 1.0 Brings Day-2 Controls to Kubernetes AI Inference

A stable release built for running inference day to day

The llm-d project shipped version 1.0 on October 20 at the PyTorch Conference North America. The project is an open-source distributed inference stack that runs natively on Kubernetes, and its backers present this release as the point where it is ready for routine production use, not only for benchmarks31. Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA founded llm-d in 2025. The Cloud Native Computing Foundation (CNCF) now governs it, and the contributors report more than 500 developers from over 25 organizations48.

The accounts of what is technically new line up closely. Version 1.0 adds four things: multi-tenant quality of service through priority- and fairness-aware flow control; zero-downtime operations through a high-availability router and rolling updates; a stable v1 configuration API; and "agent-aware" serving that keeps multi-turn sessions close to their cached context58. The capabilities that made the project's name were already shipping before this release. Those include prefix-cache-aware routing, KV-cache offloading, prefill/decode disaggregation and wide expert parallelism6.

Carlos Costa of IBM Research, an llm-d co-founder and its lead maintainer, gave the shortest summary. In his telling, 1.0 is about "day 2 operations": a stable API, tenants with competing priorities sharing one fleet, and upgrades that don't interrupt service3. That framing is the story. 1.0 is less a performance release than a release about whether operators can run the stack.

Why the operational features matter more than the benchmarks

Raw speed has rarely been the hard part of self-hosted LLM serving. Running it like any other critical service has been. Generic Kubernetes load balancing spreads requests across pods round-robin, which breaks up the KV caches that make LLM inference efficient19. llm-d's router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas already hold a given cached prefix, then weighs load and predicted latency before choosing where a request goes1.

The new 1.0 features deal with what happens once that system is live. Flow control lets latency-sensitive interactive traffic and bursty agent workloads share a fleet without one starving the other5. That lets a platform team run one pool of accelerators for both kinds of traffic, instead of building a separate cluster for each13. A new component called DisaggregatedSet manages prefill and decode pools as a single unit, so rolling updates don't drop context or break active sessions1. It builds on LeaderWorkerSet, the Kubernetes API for multi-host inference1. This matters because disaggregation has been one of the stack's headline optimizations, and splitting a model across two pools of replicas makes upgrades far more fragile. Lockstep rollouts are what make that split workable in practice5.

The stable API matters just as much for adoption. Analysts at NAND Research note that the v1 configuration surface brings flow control into one place, and that the release materials list no breaking changes or migration steps13. For enterprise platform teams, a 1.0 API is the signal that it is safe to standardize on.

Agents are the workload the release was built around

Every account of the release puts agentic systems at the center. NAND Research describes a cost problem in which long, repeatedly reused contexts make input processing the largest expense13. Coding agents and multi-step tools send the same growing context back on every turn. Routing each turn to the replica that already holds that context, instead of recomputing it, is where most of the savings come from.

In 1.0, agent-aware serving works in three layers:

• A session-state plugin tags requests so follow-ups go to the same backend until that replica saturates13.
• Turn-priority scheduling weighs where a request sits in an ongoing exchange13.
• Native Responses API state carries context across turns5.

There is also a new asynchronous processor, llm-d-async-processor. It pulls latency-tolerant jobs from queues such as Google Cloud Pub/Sub or Redis and runs them in spare capacity, using strict priority lanes and quotas so interactive users are not crowded out1.

Google's keynote demo made the agent focus concrete. It showed 1,000 sandboxed agents hitting a single llm-d endpoint that spanned vLLM, SGLang, and a mixed fleet of TPUs and GPUs1. IBM Research and Red Hat ran a larger test with GLM-5.2, a mixture-of-experts model of about 753 billion parameters, on 544 NVIDIA H100 GPUs. It peaked above 6.6 million output tokens per minute and served up to 3,000 concurrent coding agents with zero preemptions13. NAND Research adds a detail the press releases leave out: 3,000 agents was close to the system's limit, and 2,500 was the comfortable operating point13.

The project also reaches into training. Google says accelerators often sit idle 40% to 60% of the time during reinforcement-learning rollouts. llm-d's time-slicing fills those gaps with async inference, lifting duty cycles from about 40% to above 70%1. MarkTechPost points out that this still leaves roughly 30% of capacity idle6.

Weighing the numbers

The performance claims are large, and nearly all of them come from vendors. Google reports these results:

• Ironwood TPU coding workload: median time-to-first-token (TTFT) fell by more than 85% and throughput rose 2.4x compared with a plain Kubernetes Service1.
• Gemini platform: the prefix cache hit rate doubled from 35% to 70%1.
• 17,000-node multi-region deployment: throughput scaled 2.9x across three clusters with a 99.9% success rate1.

Red Hat's release headlines cost cuts of five to ten times for context-heavy agentic workloads, plus three times higher output throughput4. The cost figure is a comparison of per-token costs at current cloud rental rates against equivalent commercial API pricing, and the savings are largest for input-heavy traffic813. IBM Research also reports an 11.5x cost advantage over an all-proprietary setup for a quantum error-correction hardware design task. That workload ran at about $0.11 per million tokens with a prefix-cache hit rate near 98%20.

Skepticism is warranted, and the more careful coverage says so. Runtimewire notes that the cost and throughput claims are tied to tests the companies ran themselves10. NAND Research advises treating the five-to-ten-times claim as a hypothesis to test against your own traffic, since results depend on context reuse, GPU utilization and how well open-weight models perform on the task13. A 98% cache hit rate is a best case: highly repetitive context. It is not a typical enterprise mix. The fair reading is that the mechanism is sound and the size of the gain depends on the workload.

The adoption evidence is harder to dismiss. JetBrains says llm-d runs in production behind its Junie Lite plan8. DigitalOcean says the stack underpins its Serverless Inference and Managed Agents products, which serve trillions of tokens2. Roblox describes using it across a mix of cloud and on-premises GPUs without being tied to one inference engine4. Snap reports prefix cache hit rates of up to 75% to 80%6.

Who owns the story

The framing differs depending on who is telling it. Google's announcement centers on Google Cloud TPUs and GKE Inference Gateway, which runs the llm-d router as a managed service1. MarkTechPost's headline credits Google with shipping the release6. Red Hat and IBM present it as a community milestone under CNCF governance43. Even the quotes are inconsistent: one IBM newsroom version attributes a statement to Red Hat's Brian Stevens, while Red Hat's own release credits CNCF CTO Chris Aniszczyk with the same words24.

This is more than housekeeping. Neural Magic created llm-d, and Red Hat acquired that company in 202511. The project was donated to the CNCF as a sandbox project in March 2026 at KubeCon Europe11. Every founding company has a commercial angle. Red Hat sells OpenShift AI and Red Hat AI Inference. Google wants GKE workloads. CoreWeave offers llm-d alongside NVIDIA's own Dynamo stack132. NAND Research reads the shared ownership as a strength: competing hardware vendors are steering the same project13. One commentary puts the stakes plainly. If serving, not training, is where AI's money goes, then whoever controls the serving layer controls the bill14.

There is competition as well. MarkTechPost lists NVIDIA Dynamo, which supports TensorRT-LLM but only NVIDIA GPUs, and AIBrix, which started at ByteDance6. llm-d's distinguishing traits are neutral governance and support across NVIDIA, AMD and Intel GPUs as well as Google TPUs6.

The takeaway for teams running inference

The best reading is that llm-d 1.0 marks the moment Kubernetes-native inference moved from optimization research into platform engineering. Red Hat's Robert Shaw said in March that the next phase would bring multi-tenant serving, request prioritization and closer alignment with the security needs of agentic systems16. Seven months later, the first two have shipped.

Google's suggested adoption path is incremental. Teams can start with prefix-aware routing in front of an existing vLLM deployment, then add flow control, KV-cache offload and disaggregation1. That lowers the cost of trying it. For organizations running open-weight models at scale, especially agent-heavy ones, the stable API and broad vendor backing make a pilot reasonable now13. Whether the cost savings hold up should be judged on their own traffic, not on launch-day benchmarks.