Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
A new open-source engine claims to run the full 2.78T-parameter Kimi K3 model locally on a laptop using about 29 GB of RAM by streaming most weights from SSD, achieving roughly 0.5 tokens per second. Commenters are split on its practicality at that speed and energy cost, but see it as an interesting proof of what’s technically possible for local, high-end LLM inference. Much of the debate centers on the heavy use of AI-generated code and documentation, quantization trade-offs versus “pure” models, and whether such projects prioritize craft, clarity, and long-term usability.
Project concept and performance
- Tool runs full 2.78T-parameter Kimi K3 on commodity hardware by streaming most weights from SSD, using only ~29 GB RAM for 4k context.
- Throughput is about 0.5 tokens/second; some see this as mostly a proof-of-concept rather than practical today.
- On macOS, ARM NEON was reportedly faster than Metal for this project.
Practical uses of a 0.5 tok/s K3
- Many consider 0.5 tok/s unusable for interactive work, even for email-level workflows.
- Others suggest overnight or multi-hour batch tasks: code review, project analysis, summaries of meetings or weeks of work.
- Debate over how much “thinking” a model can do at this rate; one commenter notes that models may use tens of thousands of internal tokens to produce short answers.
Storage, memory, and system design
- Discussion contrasts custom streaming engine vs. mmap-based approaches like llama.cpp.
- Some claim manual prefetching and caching can significantly outperform generic OS paging.
- Concerns about swap: several argue to disable swap entirely to avoid SSD wear and poor performance; others clarify that read-only mmap of weights does not harm SSD endurance.
Quantization and model fidelity
- This engine uses a re-quantized 3‑bit residual variant, not the “pure” K3.
- Another project (deltafin) is cited as running unaltered K3 but with higher per-token bandwidth (≈25.8 GB vs ≈17 GB).
- One commenter calls the README confusing or self-contradictory on whether the model is truly “native precision”; overall precision and quality impact are deemed unclear.
Cost and energy efficiency
- Rough cost estimate: ~$5 per million tokens at 42W and $0.20/kWh.
- Some compare this unfavorably to modern GPU clusters (orders of magnitude more tokens per Wh), but note GPUs have high upfront cost.
- Solar/PV discussion centers on whether self-consumed energy should still be treated as a real opportunity cost.
Documentation quality and LLM authorship
- Large subthread criticizes the LLM-generated README as verbose, opaque, and internally-focused.
- Others defend LLM-assisted docs as better than no docs, but agree they should be edited for clarity.
- Broader debate about using LLMs for code, commits, and docs: some see it as “lazy slop,” others as efficient collaboration if code is reviewed.
Licensing, branding, and trust
- Some distrust the company due to past non-open-source licenses and perceived name confusion with SQLite.
- The company responds that it has rights to the name and that this project will remain permissively licensed.
Related efforts and outlook
- Thread catalogs multiple near-simultaneous Kimi K3 self-hosting efforts: CPU-only streaming, consumer GPUs, and compressed GGUF variants.
- Many view this project as an early, impractical step toward eventually viable local execution of very large models.