Jamesob's guide to running SOTA LLMs locally

Original Article ↗ Hacker News Discussion ↗

Hardware Options and Performance

  • Popular “sweet spot” setups:
    • Single RTX 3090 / 4090 or similar (~24 GB VRAM) for Qwen 3.6–27B/35B at 4–6 bit with usable speeds.
    • Dual 3090s (~48 GB VRAM) praised for very high memory bandwidth and good throughput on Qwen 27B.
    • 96–128 GB VRAM boxes (RTX 6000, OEM/Spark, Jetson, Arc B70) used to run multiple models or larger contexts.
    • Apple M-series (esp. M4/M5 Max 48–128 GB) seen as convenient unified-memory boxes but significantly slower than big GPUs for large models.
  • Multi-GPU 4×/8× RTX 6000–class rigs can run REAP-pruned, heavily quantized GLM 5.2 NVFP4 at tens–100+ tok/s, but require $40k–$100k+ and complex setup.

Economics: Local vs Cloud

  • Many argue large local rigs are economically irrational versus:
    • $200/month frontier-model subscriptions.
    • Cheap per-token providers (DeepSeek, GLM, MiMo via OpenRouter, etc.).
  • Counterpoint: local is valuable when:
    • Extreme data privacy / on-prem requirements exist.
    • You want 24/7 autonomous agents without token costs.
    • You want to hedge against future price hikes or service restrictions.
  • Consensus: for most individuals, use cloud now; only buy big hardware if you already have GPUs, love tinkering, or have strict constraints.

Model Quality, Quantization, and Context

  • Heavy quantization (Q4, REAP-pruned MOEs) is widely used but:
    • Several report noticeable degradation on long-horizon coding / analysis despite good benchmarks.
    • Small/chat tasks often seem fine; deep, long-context workflows expose accumulating errors and looping.
  • Some claim modern sampling strategies (DRY, top-n-sigma, min_p, XTC) greatly mitigate long-context and quantization issues; others remain skeptical.
  • Running full-precision frontier-scale models locally is described as currently impractical (hundreds of GB–TB VRAM).

Use Cases and Harnesses

  • Local LLMs praised for: privacy, no token limits, offline work, and “good enough” coding, debugging, research, summarization, and KB maintenance with Qwen/Gemma-scale models.
  • Sophisticated multi-agent “harnesses” (Pi, Vibe, Claude Code-style setups, custom orchestrators) can extract more from smaller models but add complexity and security concerns.

Speech-to-Text and Ancillary Models

  • Whisper is no longer viewed as STT SOTA; Parakeet v3 and Voxstral cited as more accurate/faster.
  • Local STT + LLM assistants are considered practical on modest VRAM.

Security and Isolation

  • Strong advice to sandbox agents (VMs, microVMs, OS sandboxes, SELinux, restricted FS access).
  • GPU passthrough VMs are seen as trickier; some prefer LLM on host, agents in isolated guests.