Groq CEO: 'We No Longer Sell Hardware'

API access, rate limits, and monetization

• Multiple developers report very low free-tier limits (≈30 requests/min, dropping to 10 with “all day” usage), calling them unusable for anything beyond toy projects.
• Some complain they cannot even add a credit card; billing has been “coming soon” for a long time, leading to speculation they can’t service demand or don’t want small customers.
• Reliability concerns: users see periods of extremely fast responses followed by long stalls (10–30s), making it hard to use in production.

Shift from Hardware Sales to Cloud Service

• Many see the “no longer selling hardware” move as driven by the difficulty and cost of supporting customers on a novel architecture (hardware failures, QA, training, integration).
• Running Groq-owned datacenters and selling access is viewed as higher-margin and operationally simpler than shipping and supporting cards.
• Some interpret this as positioning for acquisition or a “shadow acquisition,” though this is unverified.

Architecture, Performance, and Scaling

• Groq’s appeal: extremely fast inference (often 400+ tokens/s, very low latency) on relatively old 14nm silicon, without HBM, using lots of SRAM plus a static, ahead-of-time compiler.
• Commenters compare it to a simplified, deterministic take on VLIW, well-suited to tensor workloads.
• However, the architecture scales poorly downwards: claims that tens to hundreds of chips are needed even for mid-size models; early public demos used ~568 LPUs.
• There are concerns about scaling to GPT‑4‑class models (may require 10k+ chips while current interconnect supports only a few hundred).

Economics, Power, and TCO

• Debate over cost-effectiveness: some argue Groq needs far more chips and overall power than GPUs (H100) for modest speed gains; others counter that reduced data movement could still make it efficient.
• High capex (many cards, racks, networking) and non-trivial failure rates for high-end hardware are emphasized as serious practical hurdles.
• Several note that custom silicon design and tapeout are extremely expensive; Groq likely needs large, high-value deals rather than small card sales.

Latency, Quality, and Use Cases

• Enthusiastic users say Groq is dramatically faster than mainstream LLM APIs and route most of their traffic there, arguing “latency is king,” especially for interactive or agent-like workloads.
• Others question how much sub-second vs few-second latency matters when LLMs already replace minutes of human work, and stress that model quality (closed models vs open weights) can be more important.
• Some see fast open-weight models as enabling new agent and tool-chaining architectures.

Competition and Alternatives

• Fireworks.ai is mentioned as a strong, more “honest” high-speed inference provider.
• Tenstorrent and AMD MI300X clouds are discussed as emerging hardware alternatives, with detailed talk of software stacks, reliability, and open benchmarking.
• There is speculation that Groq’s business model could be short-lived as conventional “systems-of-chips” catch up on SRAM and throughput, but others note similar “short-lived” claims have not yet doomed other AI API providers.

Saudi Aramco and Non-LLM Workloads

• A large Aramco deal is cited; some question why an oil company wants such specialized AI hardware.
• Explanations include materials discovery, reservoir simulation, and parallel numerical workloads generally.
• A Groq employee states the system was designed for arbitrary high-performance numerical computing and has been used for scientific workloads like fusion and drug discovery.