Cheaper inference from routing and caching on the same GPUs is the part of the AI capex story I actually like.

Cheaper inference from routing and caching on the same GPUs is the part of the AI capex story I actually like. llm-d 1.0 points that way, but every headline number so far comes from the project's own sponsors.

• Contributors released v1.0 of llm-d on Oct. 20, 2026. It's an open-source, Kubernetes-native distributed inference stack, founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, and now a CNCF Sandbox project.
• In an IBM demonstration, GLM 5.2 (about 753B parameters) peaked above 6.6 million output tokens per minute and served up to 3,000 concurrent coding agents without preemptions.
• IBM Research reports one workload at about $0.11 per million tokens with a ~98% prefix-cache hit rate, 11.5x cheaper than an all-proprietary configuration. IBM separately estimates 5-10x lower per-token cost versus commercial API pricing at current cloud rental rates.
• The savings come from serving infrastructure, so results depend on workload, hardware and cache behavior. A 98% hit rate is a friendly setup.

If cost per token keeps falling like this, more AI names have a path to real cash flow, which is what would push me off my mixed view on AI capex. What I'd watch next: hyperscalers outside the founding group adopting it, and whether anyone outside IBM reproduces the 11.5x.

Source:
eeNews Europe: https://www.eenewseurope.com/en/llm-d-1-0-targets-lower-ai-inference-costs/

Not financial advice. For informational and educational purposes only.