DeepInfer: High-performance inference framework for DeepSeek models

Original Article ↗ Hacker News Discussion ↗

China’s DeepSeek has open-sourced a high‑performance inference framework for its V4.1 Flash LLM, revealing that a single cluster can serve 155 trillion tokens per day and generate around $38 in revenue per GPU hour at off‑peak pricing. Commenters see this as strong evidence that LLM inference can be highly profitable and increasingly commoditized, especially when paired with cheap Chinese power and Huawei NPUs, but question whether those margins can survive intense competition, falling prices, and ongoing training costs. Many argue that such open tooling erodes moats for US frontier labs, strengthens open‑weight and self‑hosted models, and may shift AI power toward infrastructure owners and hardware vendors rather than model providers alone.

DeepInfer release & what it is

  • DeepSeek open-sourced “DeepInfer”, a high‑performance inference framework optimized for their V4.1 Flash model and for NVIDIA SM100 and Huawei Ascend 950 NPUs.
  • It’s built on ideas from vLLM / SGLang but adds DeepSeek’s own optimizations; some critical pieces for production (KV cache system, full load‑balancing, VE/prefill/decode disaggregation) are not yet included.
  • Several commenters expect its techniques to be upstreamed into common OSS engines, benefiting the entire ecosystem, including US labs.

Inference economics & profitability

  • DeepSeek claims ~$38/hour revenue per card at off‑peak pricing; a cluster reportedly handles ~155T tokens/day.
  • Many argue this shows inference itself is highly profitable and that GPUs can pay back in under a year, especially with high utilization and cheap Chinese power.
  • Others counter that “inference profitability” is misleading without including massive training, R&D, and debt costs; overall lab P&Ls may still be negative.
  • Debate over long‑term margins: some see inevitable race‑to‑the‑bottom commodity margins; others think frontier “luxury intelligence” will retain high pricing.

Commoditization, moats, and competition

  • Strong view that generic LLM inference is an undifferentiated commodity: many providers, price‑sensitive buyers, low switching costs, open weights.
  • That threatens US frontier labs’ trillion‑dollar narratives; their only moats may be brand, distribution, proprietary data, or regulatory capture.
  • Counter‑view: frontier quality still matters; many enterprises will pay premiums for best‑in‑class models and compliant, “trusted” providers.

China vs US, hardware & regulation

  • DeepSeek’s openness is seen as both economic warfare (undercutting US labs) and as a way to work around GPU export controls by having Western DCs host Chinese open‑weights.
  • Huawei Ascend support is notable; some see this as marketing and a challenge to NVIDIA’s dominance, especially given China’s cheap electricity and large power buildout.
  • A new US DHS order bans DeepSeek products from federal civilian systems (and some third‑party implementations), seen as the start of formal decoupling.
  • European context is mixed: concerns about both US and Chinese control, but strong interest in self‑hosted, open‑weight models for sovereignty and GDPR reasons.

Technical notes & practical impact

  • DeepSeek’s KV cache strategy (retention ≥72 hours, very compact per‑token footprint) is viewed as a major efficiency win; the actual implementation is not fully open yet.
  • Many expect smaller inference providers to adopt DeepInfer to narrow the performance gap with DeepSeek’s own API, driving token prices lower.
  • Some developers still prefer big‑lab subscriptions (flat monthly cost, stronger quality) over DeepSeek’s pay‑per‑token API despite its low unit price.