Groq runs Mixtral 8x7B-32k with 500 T/s

Original Article ↗ Hacker News Discussion ↗

Groq has unveiled an LLM hosting service where the Mixtral 8x7B model runs at roughly 500 tokens per second on custom “Language Processing Unit” ASICs, dramatically outpacing typical GPU-backed deployments. Commenters are impressed by the near-instant, GPT‑3.5–class responses and low per‑token pricing, but probe how this is achieved, questioning total hardware cost, power use, scalability beyond a few flagship models, and long‑term sustainability of the business model. The thread also explores technical details such as deterministic, SRAM-heavy architecture, partial FP8 quantization, support for fine‑tuned and non‑LLM workloads, and what such latency reductions could enable for real-time voice, agents, and other interactive applications.

Performance and User Impressions

  • Many users report 400–500+ tokens/s on Mixtral 8x7B and ~200 t/s on Llama 2 70B via the Groq demo and API.
  • Subjective reactions are very positive: “instant” or “superhuman” speed, especially compared to GPT‑3.5/4 and Gemini.
  • Several say this speed materially changes UX: easier to skim, iterate, and build interactive tools; slow autoregression now feels “antiquated.”
  • Some find model quality comparable to or better than GPT‑3.5 for coding and general tasks, though not at GPT‑4 level.

Hardware and Architecture

  • Groq uses custom ASICs (“GroqChips” / LPU) with ~230 MB on‑chip SRAM each, built into racks of hundreds of chips; current demo reportedly uses ~568 chips.
  • Architecture emphasizes determinism, fixed clocking, and a systolic-like matrix engine with deterministic chip‑to‑chip interconnect (~100 Gbps links).
  • KV cache is used; decoding currently runs effectively at batch size 1, with high throughput due to the architecture.
  • Some compiler pipeline and infra are implemented in Haskell; other components use C++/MLIR and Python.

Latency, Throughput, and Batching

  • Groq positions itself as optimized for low-latency, small-batch inference, as opposed to GPU stacks that rely heavily on large batching to hit high throughput.
  • Independent benchmarks cited show very low time‑to‑first‑token and high single‑request throughput, including via the public API.
  • Discussion notes that GPUs can achieve very high total tokens/s, but typically at lower per‑user speed.

Precision, Quantization, and Quality

  • Activations are computed in FP16; some weights are stored as FP8 “at rest.” Groq claims no meaningful quality loss vs full FP16.
  • Users report Mixtral quality at least on par with strong 3.5‑class models; others highlight clear hallucinations and arithmetic failures (e.g., primes, factorization), emphasizing that faster hardware doesn’t fix model limits.

Cost, Scale, and Business Model

  • Retail accelerator cards are expensive (~$20k each), implying multi‑million‑dollar hardware for the demo scale; Groq says its own cost is much lower than retail.
  • Some observers question tokens/s per dollar and total power draw; others note that low latency may justify high capex for certain use cases.
  • Groq sells both tokens “as a service” (with a promise to beat competitors’ per‑token prices for supported models) and full systems to enterprises; no near‑term hobbyist hardware is planned.

API, Ecosystem, and Access

  • There is an OpenAI‑style chat/completions API with partial compatibility; docs and a Discord are provided.
  • Massive recent interest has created waitlists and queuing; some users report long waits despite fast generation once started.
  • Multiple developers express strong interest in using Groq as a backend for apps, RAG systems, and code tools.

Use Cases and Applications

  • Low latency is seen as especially valuable for: real‑time voice agents (CNN demo), call centers, trading, “copilot”‑style assistants, games/VR NPCs, and multi‑step agent pipelines.
  • Groq claims the hardware is general‑purpose numerical compute, suitable for vision and diffusion models; some diffusion work exists but isn’t publicly exposed yet.

Critiques, Limitations, and Open Questions

  • Concerns include: lack of HBM limiting model capacity per chip; need for hundreds of chips per large model; unclear total system throughput; and potential unsustainable pricing.
  • Some argue GPUs with HBM may be better for multi‑model, multi‑LoRA, or on‑prem scenarios; others counter that Groq’s deterministic system and SRAM are critical for ultra‑low latency.
  • Bugs and UX issues are noted: confusing model‑switching behavior in the demo, mobile UI quirks, font-loading dependencies, and occasional looping outputs.
  • Several users stress that hallucinations and reasoning limits remain; speed alone doesn’t solve core model reliability issues.