Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

Original Article ↗ Hacker News Discussion ↗

Running a 26B-parameter Gemma 4 model on 10–13-year-old Xeon CPUs at around 5 tokens per second proves that large modern LLMs can run locally on aging hardware, given enough RAM and careful quantization. Commenters weigh how practical this really is, contrasting slow token speeds and high power draw with the appeal of privacy, unlimited usage, and “free if you already own it” compute, and comparing costs against cloud inference providers. The thread also touches on the trajectory toward ever-larger sparse/MoE models on consumer machines, and raises concerns about AI-generated technical blog posts and community norms around disclosure.

AI-Written Content & HN Norms

  • Several commenters argue the blog post and some comments “read like” AI output and note that AI-written posts/comments violate HN norms.
  • Others disagree or say it’s hard to distinguish when people heavily use LLMs to draft text.
  • The author states that the code patch was AI-assisted but the post clarifies which parts were human vs AI.

Bug & Technical Details

  • The original fork assumed AVX2; older Ivy Bridge Xeons lack this, causing build failures and, more subtly, missing dispatch paths for two MoE ops.
  • On non‑AVX2 builds, MoE expert outputs came from uninitialized memory, leading to fluent but nonsensical text.
  • A fix is submitted upstream as a pull request.

Performance, Quantization & RAM

  • Multiple reports of running Gemma 4 and other models on old Xeons, dual-CPU servers, and Mac hardware.
  • Token rates vary widely: ~5 t/s for Gemma 4 26B on 2013-era Xeons; higher rates on GPUs and smaller/strongly quantized models.
  • Discussion of Q4 vs Q8: Q4 halves bandwidth needs and can nearly double speed on bandwidth-limited systems, but Q8/Q6 are preferred when RAM allows due to quality.
  • Large RAM (80–100+ GB used by 26B models) is common; some experiment with extremely low-RAM setups and custom loaders.

Usability of Slow Local Models

  • Strong split: some find 5–10 t/s acceptable for background or batch workflows; others consider it unusable for interactive coding or long reasoning chains.
  • Debate over “flow state”: fast models (hundreds of t/s) let users iterate quickly; slow models push people toward queue-and-walk-away usage.

Cost, Power & Efficiency

  • Several back-of-envelope calculations suggest local CPU inference can cost more in electricity than cloud tokens, especially with 300–500W servers and high energy prices.
  • Counterpoints: cheaper electricity, solar, or using waste heat for space heating can shift the calculus.
  • GPU power-capping can significantly reduce energy use with modest speed loss.
  • Some note that cloud providers may currently sell tokens below true cost, so prices might rise.

Privacy, Control & Motivation for Local

  • Many emphasize privacy, independence, and freedom from vendor limits as primary reasons for local inference, not cost.
  • Others see local hardware as valuable experimentation, but not yet competitive with top cloud models in quality or speed.

Future of Local LLMs

  • Optimistic predictions: >200B MoE and even ~1T-parameter-equivalent models on consumer hardware by ~2027–2028, aided by ternary/1-bit training, sparse MoE, new GPUs, and specialized accelerators.
  • Skeptics highlight RAM bandwidth, power constraints, fab capacity, and economic incentives to prioritize datacenters.
  • Some argue transformers are structurally ill-suited to efficient local inference; others see ongoing architectural and hardware work as promising.

Tooling & Configuration Issues

  • Practical tips: adjusting timeouts in Ollama, exploring LM Studio TTL/eviction settings, and using batch/agent frameworks to better exploit slow local models.