Groq CEO: 'We No Longer Sell Hardware'
Groq’s move to stop selling its AI accelerator hardware and focus on running its own low‑latency inference cloud is drawing both enthusiasm and skepticism. Commenters praise the company’s speed on open‑weights models and novel SRAM‑heavy architecture, but question its tiny free‑only rate limits, lack of billing, high apparent cluster costs, and bold claims about matching hyperscaler compute. Many see the shift as a way to avoid the support burden of selling chips directly, with some speculating about an eventual acquisition and others arguing that enterprises will ultimately prefer on‑prem or larger vendors for serious workloads.
API access, rate limits, and monetization
- Multiple developers report very low free-tier limits (≈30 requests/min, dropping to 10 with “all day” usage), calling them unusable for anything beyond toy projects.
- Some complain they cannot even add a credit card; billing has been “coming soon” for a long time, leading to speculation they can’t service demand or don’t want small customers.
- Reliability concerns: users see periods of extremely fast responses followed by long stalls (10–30s), making it hard to use in production.
Shift from Hardware Sales to Cloud Service
- Many see the “no longer selling hardware” move as driven by the difficulty and cost of supporting customers on a novel architecture (hardware failures, QA, training, integration).
- Running Groq-owned datacenters and selling access is viewed as higher-margin and operationally simpler than shipping and supporting cards.
- Some interpret this as positioning for acquisition or a “shadow acquisition,” though this is unverified.
Architecture, Performance, and Scaling
- Groq’s appeal: extremely fast inference (often 400+ tokens/s, very low latency) on relatively old 14nm silicon, without HBM, using lots of SRAM plus a static, ahead-of-time compiler.
- Commenters compare it to a simplified, deterministic take on VLIW, well-suited to tensor workloads.
- However, the architecture scales poorly downwards: claims that tens to hundreds of chips are needed even for mid-size models; early public demos used ~568 LPUs.
- There are concerns about scaling to GPT‑4‑class models (may require 10k+ chips while current interconnect supports only a few hundred).
Economics, Power, and TCO
- Debate over cost-effectiveness: some argue Groq needs far more chips and overall power than GPUs (H100) for modest speed gains; others counter that reduced data movement could still make it efficient.
- High capex (many cards, racks, networking) and non-trivial failure rates for high-end hardware are emphasized as serious practical hurdles.
- Several note that custom silicon design and tapeout are extremely expensive; Groq likely needs large, high-value deals rather than small card sales.
Latency, Quality, and Use Cases
- Enthusiastic users say Groq is dramatically faster than mainstream LLM APIs and route most of their traffic there, arguing “latency is king,” especially for interactive or agent-like workloads.
- Others question how much sub-second vs few-second latency matters when LLMs already replace minutes of human work, and stress that model quality (closed models vs open weights) can be more important.
- Some see fast open-weight models as enabling new agent and tool-chaining architectures.
Competition and Alternatives
- Fireworks.ai is mentioned as a strong, more “honest” high-speed inference provider.
- Tenstorrent and AMD MI300X clouds are discussed as emerging hardware alternatives, with detailed talk of software stacks, reliability, and open benchmarking.
- There is speculation that Groq’s business model could be short-lived as conventional “systems-of-chips” catch up on SRAM and throughput, but others note similar “short-lived” claims have not yet doomed other AI API providers.
Saudi Aramco and Non-LLM Workloads
- A large Aramco deal is cited; some question why an oil company wants such specialized AI hardware.
- Explanations include materials discovery, reservoir simulation, and parallel numerical workloads generally.
- A Groq employee states the system was designed for arbitrary high-performance numerical computing and has been used for scientific workloads like fusion and drug discovery.