Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone

Original Article ↗ Hacker News Discussion ↗

Running massive language models on consumer devices may be impractical today, but many see experiments like Swiftlet—streaming an 80B-parameter Qwen model from SSD on a Mac and a 35B model on an iPhone—as important steps toward that future. Commenters debate performance bottlenecks (especially memory bandwidth and prefill), SSD wear, and physical limits, while others argue that ongoing advances in hardware, quantization, and model architectures will eventually make powerful local AI both feasible and common. The thread also contrasts centralized cloud inference with on-device models, highlighting trade‑offs in cost, latency, privacy, and user control.

Performance and Practicality

  • Several commenters note that decode token/sec is only part of the story; prefill becomes the bottleneck, so large prompts can take many minutes or more (e.g., ~30 minutes for 10k tokens on an M5).
  • Current speeds are seen as too slow for coding or interactive agents, but potentially acceptable for background or overnight batch jobs.
  • Some argue it may be cheaper or simpler to rent a GPU briefly than leave a desktop-class machine running for hours.

Storage, SSD Wear, and Memory Bandwidth

  • Concerns about “killing” SSDs through heavy streaming are debated.
    • Consensus: writes are the main issue; read wear (read disturb) exists but is likely not a practical problem for non-24/7 full-drive workloads.
    • Others report long-lived SSDs and little real-world failure.
  • Multiple comments stress that memory bandwidth, not raw FLOPs, is the key constraint for local LLMs, especially when weights sit on SSD.

Value of This Work and Long-Term Trajectory

  • Many see such projects as foundational: rough, impractical early steps that enable later breakthroughs and “motivated amateurs.”
  • Skeptics argue this is like “climbing a tree to reach the moon” and can’t realistically lead to 1T-parameter models on cheap SSDs.
  • Optimists counter with historical hardware gains and expect both hardware and software to keep improving, though physical and bandwidth limits are acknowledged.

Centralized vs On-Device Inference

  • One camp expects 99%+ of usage to stay centralized due to efficiency, parallelization, and user habits.
  • Others emphasize privacy, offline capability, and “good enough” small models as strong drivers for on-device and on-prem setups.

Hardware Futures: GPUs, Apple, ASICs, FPGAs

  • Discussion of GPU FLOPs and memory bandwidth trends suggests steady but not explosive gains; memory capacity lags FLOPs.
  • Apple’s unified memory and NPUs are cited as well-suited to co-designed on-device LLMs; some speculate about LLM-optimized accelerators or even models baked into silicon.
  • There is mention of existing “model-on-silicon” products and speculation that private actors already do this for niche, latency-critical domains.

Model Architectures and RAM Tuning

  • MoE and sparsity are seen as key to fitting large models into limited RAM, but selecting which experts stay “hot” in memory is nontrivial.
  • The project exposes a tunable RAM cache; current bottleneck is GPU dispatch rather than SSD, so extra RAM helps less than expected for now, but may matter as kernels improve.

Safety, Misuse, and Web Access

  • Some worry that democratized powerful local models will ease hacking and scams; others argue defenses will also improve with the same tech.
  • For web-aware local agents, commenters point to search APIs or self-hosted meta-search as realistic options; downloading a full web snapshot is deemed impractical.

Project Details and Attribution

  • The “built in collaboration with Claude Code” line is clarified as meaning the author used an AI coding assistant heavily, not that there was formal involvement from that vendor.
  • Prior similar “first time on a phone” claims for streaming-weight setups are noted, so the uniqueness of this milestone is seen as somewhat unclear.