Show HN: Getting GLM 5.2 running on my slow computer

Original Article ↗ Hacker News Discussion ↗

Project concept and hacker spirit

  • Tool runs very large GLM 5.2 MoE models on “slow” / consumer hardware by aggressively streaming weights from NVMe instead of holding everything in RAM.
  • Many comments praise the minimalist, almost obfuscated implementation and the general “run things where they weren’t meant to be run” spirit.
  • Several people mention doing similar experiments (custom streaming engines, mmap-based runners, Medusa/MTP, Rust ports) and see this as part of a broader trend.

Performance and practicality

  • Reported speeds for low-end hardware (≈0.05–0.1 tok/s) are seen by some as too slow for interactive use, though others find even 1 tok/s acceptable for overnight or batch workloads.
  • There is interest in benchmarks on higher-end CPUs, high-RAM servers, and GPUs; some expect that with enough RAM a standard llama.cpp setup might outperform SSD streaming.
  • Comparisons arise with other projects (llama.cpp, flash-moe, hypura, fastllm), and one open question is whether this approach is actually faster than mature engines that already use mmap and quantization.

Hardware, affordability, and alternatives

  • Debate over whether “ordinary people” can afford 24+ GB RAM and large NVMe drives, given rising memory prices.
  • Suggestions include used rackmount servers with 512GB–1TB RAM, Mac Studio–class machines, or cloud-hosted GLM 5.2 as often more practical.
  • Multiple users explore ideas like RAID0/parallel SSDs, hybrid CPU/GPU setups, and exploiting unified memory on Apple Silicon.

SSD wear, swapping, and safety

  • The README warns about potential SSD wear; this triggers concern for laptops with soldered SSDs and proposals to use external “burner” drives.
  • Explanations vary: some attribute risk to swap and page cache eviction (“spilling”); others stress the app itself does only reads and that the warning is conservative.
  • Mitigations discussed: disabling swap, tuning swappiness, read-only partitions or images. The exact magnitude of wear from heavy reads remains somewhat unclear.

Interfaces and workflows for slow models

  • Several commenters argue chat UIs are a poor fit for very slow local models and propose ticketing/job systems instead.
  • Ideas include orchestrators that dispatch tasks to worker agents, and simple integrations via email and ticket queues to treat the model like a slow but reliable teammate.

MoE- and architecture-specific questions

  • Questions raised about distributing experts over MPI in a cluster; answer: theoretically possible, but network latency likely dominates on small clusters.
  • Disabling all but one expert is deemed possible but would drastically degrade quality; using a smaller dense model is recommended instead.