llm-d 1.0 makes self-hosted AI inference a credible production option
The open Kubernetes stack now covers the operational layer that model servers alone leave behind, but owning inference still only pays when traffic or constraints justify the burden.
What shipped
Released on October 20, llm-d 1.0 is an open-source, Kubernetes-native control plane for distributed model serving. It sits above engines such as vLLM and SGLang, adding stable APIs, cache- and load-aware routing, multi-tenant flow control, high-availability routing, coordinated upgrades and session-aware handling for agent workloads.
That is materially different from another faster model server: it targets the parts that usually turn self-hosting into an operations project.
Why operators should care
Open-weight models were already runnable; the hard question was whether a team could keep them reliable under mixed traffic without rebuilding a cloud provider’s serving layer. llm-d 1.0 narrows that gap by treating context reuse, bursty agents, priority traffic and rolling changes as first-class production concerns.
It does not make capacity planning, accelerator procurement, observability or incident response disappear. The software lowers platform lock-in while transferring more uptime and utilization risk to the operator.
The build-or-wait decision
For teams with sustained volume, data-residency requirements or existing accelerator fleets, this is worth a controlled pilot now. Put llm-d behind a provider-neutral gateway, replay real production traces, and compare cost per completed task, tail latency, cache reuse, failure recovery and operator time against managed APIs.
For small or volatile workloads, we would wait. Version 1.0 makes self-hosting credible; it does not make idle hardware or a thin platform team economical.
Virtual Arc’s view is that llm-d 1.0 is the first release in this wave that changes self-hosting from a model demo into a defensible platform decision, but only for teams that already have enough traffic or constraints to deserve a platform. We would not migrate because an open model looks cheaper per token; token price ignores cache misses, spare capacity, rollouts, on-call load and failed tasks. We would first shadow-route production traffic through llm-d and at least one managed API, hold the application contract stable, and score cost per completed task, p95 latency, recovery behavior and operator effort. If the open stack wins under real traces, we would move predictable workloads first and keep a managed fallback. If it does not, we would keep buying inference. The controversial part is simple: 1.0 is a reason to test now, not a reason to own GPUs by default.
Turn insight into a working system.
We would build a provider-neutral inference layer, replay your production traffic against llm-d and managed APIs, and add routing, permissions, billing and audit controls before any migration. Talk to us if your team runs sustained agent workloads or faces data-residency constraints.