NVIDIA’s Groq 3 LPX enters production: 3,400 tokens per second is not a product yet
The hardware benchmark could reset latency expectations for agentic systems, but without a public endpoint, price or availability date, production teams should prepare to test—not migrate.
What entered production
On August 24, NVIDIA said its rack-scale Groq 3 LPX systems had entered full production. In a benchmark conducted by Artificial Analysis on NVIDIA-hosted hardware, Gemma 4 31B generated 3,431 output tokens per second with 100,000 tokens of input context, roughly four times the nearest alternative platform.
Nebius is named as the first AI cloud adopter, while Groq also plans to deploy LPX alongside Vera Rubin NVL72. Neither announcement gives builders a public API date or price yet.
Where speed changes the economics
Waiting compounds inside an agent loop: a system plans, calls tools, checks results and revises through multiple sequential generations. Faster decoding can therefore cut total wall-clock time even when model quality remains unchanged.
But headline tokens per second are not cost per completed task. Queueing, context processing, external tools, failures and retries can erase an impressive laboratory advantage.
What we would do
Virtual Arc would not redesign around LPX today. We would preserve a provider-neutral routing layer, instrument queue, prefill, decode, tool and retry time separately, and prepare a fixed evaluation set of long-context, output-heavy workflows.
Once a real endpoint, pricing and operating terms appear, we would run a narrow production canary. We would route workloads to LPX only if both p95 completion time and cost per successful task improve without a reliability regression.
Virtual Arc’s view is that this is a capacity signal, not a migration trigger. The 3,431-token-per-second result matters because decode latency is one of the few bottlenecks that multiplies across every step of a long agent loop; if it survives real APIs, mixed workloads and concurrency, some workflows that feel batch-like today could become interactive. But the benchmark covers one model on NVIDIA-hosted hardware, while teams still lack a public LPX endpoint, price, specific SLA and independent production data. We would therefore spend engineering time on observability and portability now, not on an LPX-specific architecture: separate prefill, decode, queue, tool and retry time; preserve a provider-neutral routing layer; and prepare a fixed evaluation set of long-context, high-output tasks. Once public capacity appears, we would canary only those workloads and promote LPX traffic only if p95 completion time and cost per successful task both improve without a reliability regression. Raw tokens per second do not pay the bill; completed, correct work does.
- With Groq 3 LPX in Full Production, NVIDIA Extends Vera Rubin Inference for Agents
- How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin
- Groq Among the First to Bring NVIDIA Groq 3 LPX and Vera Rubin NVL72 to Market
- Nvidia’s dedicated inference accelerator Groq 3 LPX enters full production to supercharge AI agents