Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

Original Article ↗ Hacker News Discussion ↗

A developer has managed to run Kimi K3, a 2.8‑trillion‑parameter Mixture‑of‑Experts model, on a MacBook Pro by streaming about 1.45 TB of expert weights from four SSDs, achieving roughly 1 token per second but with a 6‑minute delay before the first token on a 512‑token prompt. The project digs into how disk bandwidth, read scheduling, and Thunderbolt constraints shape performance, documenting which optimizations helped and which hurt (e.g., RAID‑0 striping and RAM caching). Commenters debate the practical value of such a slow, storage‑bound setup versus its significance as a proof of concept for running frontier‑scale models locally and fully offline.

Model & Performance

  • 2.78T-parameter MoE (Kimi K3) with ~1.45 TB of expert weights, run on a MacBook Pro M5 Max (128 GB) with 4 SSDs.
  • Experts are streamed from disk; attention “trunk” (~50 GB) stays resident in unified memory at int8 precision.
  • Reported decode: ~1 token/s over 512 tokens, ~1.13 tok/s over 128 tokens, with consistent behavior across tests.
  • Time-to-first-token on a 512-token prompt is ~6.3 minutes due to heavy prefill I/O.

I/O Architecture & Scaling

  • One 17.5 MB file per (layer, expert); each layer reads 16 experts per token, and must wait for the slowest read.
  • Four SSDs via Thunderbolt 5 enclosures; internal-only SSD gives ~half the 4-drive decode rate.
  • Empirical “drive ladder”: 1/2/3/4 drives ≈ 52% / 73–78% / 90–92% / 100% of 4‑drive speed; returns diminish with more drives.
  • RAID0/striping and some caching strategies were measured and found to hurt performance, mainly because barriers are set by the slowest device.

Prefill, Context, and Workload Fit

  • Prefill is the main bottleneck: current scheduling re-reads experts multiple times, causing ~9 TB of reads for a 1.4 TB model.
  • This makes “single-token classifier” workloads (long prompt, 1 token out) currently the worst case, not the best.
  • KV cache grows ~2.8 MiB per token; with 128 GB RAM and other reservations, context is capped around 4.4k tokens on this setup.
  • A proposed “expert-major” prefill (read each expert once per layer, process all routed rows) could cut amplification from 6.2x toward ~1x.

Use Cases & “Why Bother?” Debate

  • Skeptics argue 1 tok/s with multi-minute prefill is impractical and expensive for interactive use.
  • Supporters see value in:
    • Scheduled, unattended local jobs (reports, reconciliations) where latency doesn’t matter.
    • Proof-of-concept that frontier-like models can run locally at all.
    • “Because it’s hard” hacker experimentation and a stepping stone to more efficient designs.

Tooling, Methodology, and Documentation

  • Custom instrumentation (per-device read monitor, barrier tracing, config assertions) surfaced non-obvious bottlenecks; several >8–14% gains came from fixing these.
  • A large catalog of negative results (caching, striping, streaming the trunk, various APIs) is provided as cautionary data.
  • Some commenters find the README dense and “LLM-ish,” preferring the short TL;DR; resentment toward long, LLM-authored documentation is noted.

Hardware Futures & Model Design Ideas

  • Discussion touches on SSD bandwidth limits (Thunderbolt/PCIe), possible desktop or GPU-based setups, and ASIC-style inference chips.
  • Debate on whether future models should be more modular or “sticky” in their expert routing so only a small subset of weights needs to be memory-resident; current MoE approaches are seen as mainly saving compute, not memory.