Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac

Original Article ↗ Hacker News Discussion ↗

An open-source engine demonstrates how Google’s Gemma 4 26B MoE model can run on M‑series Macs using as little as 2 GB of RAM by streaming “expert” weights from SSD, trading raw speed for dramatically lower memory use. Commenters dissect how this approach compares to MLX, llama.cpp, and projects like Colibri and DwarfStar, noting that SSD bandwidth, OS page cache, and Apple’s unified memory are now as critical as GPU FLOPs for local inference. The thread also surfaces broader questions about how far such techniques can scale to larger MoE models, whether they’re practical for real coding work, and what they imply for future consumer hardware and on-device AI.

Overview & Goals

  • Engine runs Gemma 4 26B MoE locally on M‑series Macs using ~2 GB RAM by streaming experts from SSD instead of loading full 14 GB weights.
  • Target: low‑RAM, everyday‑task usage on Apple laptops/desktops, trading speed for footprint and offline privacy.

Performance & Hardware Dependence

  • Reported speeds:
    • ~4–6 tok/s on M1/M2 Air/Neo with 8 GB RAM.
    • ~5 tok/s on base‑SSD M4 mini; ~12 tok/s on M1 Max Studio.
    • ~31–35 tok/s on M5 MacBook Pro; ~48 tok/s on 64 GB M4 Max.
  • Large spread attributed to:
    • SSD speed (M5/M4 Max SSDs can be 3–7 GB/s vs ~2 GB/s).
    • Memory bandwidth and system‑level cache.
    • OS page cache: more RAM lets most expert weights stay cached, sharply reducing real disk reads.
  • Under heavy memory pressure, speeds drop but degrade gradually rather than collapsing.

Technical Approach (MoE + SSD Streaming)

  • Uses MoE sparsity: only a small subset of experts are active per token.
  • Implements an explicit expert cache (~16 slots by default; more slots use more RAM but improve speed).
    • Cache hit rates around 60–70%; partial reuse across 1–2 tokens.
  • Critical change: moving from mmap to parallel pread for experts:
    • Benchmarks show ~10 ms vs ~2.8–3 ms per 3.36 MB expert, and ~0.5 → ~4 tok/s on an 8 GB M2.
  • Overlaps SSD reads with GPU compute to hide latency where possible.
  • Reads 250–320 MB from SSD per token on M2 (3 GB/s during I/O phase).

Comparisons to Other Engines/Models

  • MLX with full in‑RAM Gemma 4 on M5: ~75 tok/s but ~14 GB RAM used.
  • This engine: ~31–35 tok/s on same class machine using ~2 GB RAM.
  • Discussion suggests llama.cpp with mmap‑based offload can fit in 2 GB but would be slower.
  • Other SSD‑streaming MoE engines (Colibri, Flash‑MoE, DwarfStar, MoEspresso) mentioned, generally targeting larger models and higher‑end Macs with more RAM.

Platform Scope & Limitations

  • Mac‑only: depends on Metal and unified memory; porting to Windows/Linux or discrete GPUs would require a redesign (CUDA/Vulkan).
  • Currently focused on Gemma 4 MoE; Qwen 3.6 MoE could work but architecture is more complex.
  • Not suitable for dense models or diffusion models in this implementation.
  • Very large MoEs (e.g., Kimi K3) would be theoretically runnable via similar ideas but effectively unusable on 16–64 GB machines due to extreme per‑token I/O.

Usefulness, Skepticism & UX

  • Enthusiasm: many see this as a big step for local inference on consumer Macs; impressive “26B in 2 GB” engineering; attractive for offline, privacy‑preserving use.
  • Skepticism:
    • Some argue 5–30 tok/s is still too slow versus cloud models for serious coding or interactive work.
    • Others note that high‑quality local coding assistants still demand large GPUs or frontier‑class models.
  • Gemma is viewed as strong for general tasks and multilingual basics but weaker for coding than Qwen; still “good enough” for some production use.

Future Directions & Research Ideas

  • Ideas floated:
    • Larger expert caches when more RAM is available.
    • Applying the same streaming approach to bigger MoE models as SSDs and RAM grow.
    • Speculative expert prefetching using MTP‑like heads; pushback notes this is hard because expert routing is per‑layer and depends on prior layers’ outputs.
  • Apple’s new foundation models are cited as using prompt‑level expert loading as a related but distinct approach.

Meta: Security & “LLM‑ese”

  • One commenter used an LLM to do a quick security review of the repo; others debate the value and risk of “AI says it’s safe” claims.
  • Long subthread on LLM‑polished writing style in the README:
    • Some dislike recognizable “LLM‑ese” phrases; others defend non‑native writers using tools to clean up language.
    • Consensus from several participants: code and experiments matter more than perfect prose.