Groqchat

Original Article ↗ Hacker News Discussion ↗

Groqchat showcases Meta’s Llama 2 70B model running on Groq’s custom “Language Processing Unit” hardware, achieving hundreds of tokens per second per user—far faster than typical GPU-based deployments. Commenters are impressed by the speed and low latency, especially for code and large-model inference, but note that output quality and heavy safety filters are limited by the underlying Llama 2 chat model rather than the hardware. The thread also explores Groq’s architecture and scaling strategy, its focus on inference (not training), and interest in future support for models like Mixtral and broader API access.

Performance & Benchmarks

  • Main excitement is around speed: users report ~270–300 tokens/s for Llama‑2‑70B, vs ~60 tokens/s on other services for the same model.
  • Several note this is the fastest they’ve seen for a ≥70B model; others emphasize that the point is tokens/s and latency, not model quality.
  • Some want clearer benchmarks and apples‑to‑apples comparisons (same model, context length, batch size, plus $/token and watts).
  • Groq staff mention deployments using 9 racks / 576 chips, with the entire model sharded into on‑chip SRAM for very low latency.

Model Quality, Safety, and Censorship

  • Many find answers fast but often wrong, hallucinatory, or looping; some tasks (code, basic app design advice, recipes) seem “at parity” with GPT‑4, others clearly worse.
  • Multiple users complain about strong safety filters: refusals for poetry, lyrics, light sexual/romantic content, and even benign image prompts; described as “over‑censored” or “boring.”
  • Others argue this is expected from Llama‑2‑chat style safety tuning and that model choice, not hardware, drives quality.

Hardware & Architecture

  • Groq is described as a custom‑chip hardware company; their LPUs / GroqChips form a low‑latency, high‑throughput “compute fabric” rather than GPU‑style architectures.
  • They store models entirely in SRAM across many chips (no DRAM), which is noted as expensive but ideal for fast, batch‑1 inference.
  • Training is said to still favor GPUs; Groq focuses on inference, especially where latency and tokens/s per user matter.

Ecosystem, Products, and Pricing

  • An API and pricing are “coming soon,” with claims they will be price‑competitive with OpenAI but much faster.
  • Users ask about: mixtral‑8x7B and Mistral‑7B deployments (in progress), long‑context, home/LLM boxes, single cards, and vertical use cases (e.g., healthcare).
  • Some worry about software‑stack maturity and long‑term adaptability to new architectures (e.g., MoE); responses assert the toolchain can compile arbitrary PyTorch/ONNX models.

UX & Miscellaneous

  • Several praise responsiveness but criticize the chat UI (streaming that constantly scrolls, disabled iOS autocorrect).
  • Some confusion exists around verification that outputs exactly match Llama‑2‑70B elsewhere; this remains unclear in the thread.