Jamesob's guide to running SOTA LLMs locally

Original Article ↗ Hacker News Discussion ↗

Rising interest in running state-of-the-art large language models locally is colliding with harsh economic and technical realities. Commenters weigh the appeal of privacy, control, and 24/7 availability against the high cost of multi‑GPU rigs, the compromises of aggressive quantization, and the fact that $2–40K hardware often underperforms relatively cheap cloud APIs. Many advocate a pragmatic middle ground: use modest local setups (single 24–32GB GPUs or high‑RAM Macs) and small models for everyday tasks, while relying on rented GPUs or commercial services for frontier‑level capabilities until hardware prices and model efficiency improve.

Hardware Options and Performance

  • Popular “sweet spot” setups:
    • Single RTX 3090 / 4090 or similar (~24 GB VRAM) for Qwen 3.6–27B/35B at 4–6 bit with usable speeds.
    • Dual 3090s (~48 GB VRAM) praised for very high memory bandwidth and good throughput on Qwen 27B.
    • 96–128 GB VRAM boxes (RTX 6000, OEM/Spark, Jetson, Arc B70) used to run multiple models or larger contexts.
    • Apple M-series (esp. M4/M5 Max 48–128 GB) seen as convenient unified-memory boxes but significantly slower than big GPUs for large models.
  • Multi-GPU 4×/8× RTX 6000–class rigs can run REAP-pruned, heavily quantized GLM 5.2 NVFP4 at tens–100+ tok/s, but require $40k–$100k+ and complex setup.

Economics: Local vs Cloud

  • Many argue large local rigs are economically irrational versus:
    • $200/month frontier-model subscriptions.
    • Cheap per-token providers (DeepSeek, GLM, MiMo via OpenRouter, etc.).
  • Counterpoint: local is valuable when:
    • Extreme data privacy / on-prem requirements exist.
    • You want 24/7 autonomous agents without token costs.
    • You want to hedge against future price hikes or service restrictions.
  • Consensus: for most individuals, use cloud now; only buy big hardware if you already have GPUs, love tinkering, or have strict constraints.

Model Quality, Quantization, and Context

  • Heavy quantization (Q4, REAP-pruned MOEs) is widely used but:
    • Several report noticeable degradation on long-horizon coding / analysis despite good benchmarks.
    • Small/chat tasks often seem fine; deep, long-context workflows expose accumulating errors and looping.
  • Some claim modern sampling strategies (DRY, top-n-sigma, min_p, XTC) greatly mitigate long-context and quantization issues; others remain skeptical.
  • Running full-precision frontier-scale models locally is described as currently impractical (hundreds of GB–TB VRAM).

Use Cases and Harnesses

  • Local LLMs praised for: privacy, no token limits, offline work, and “good enough” coding, debugging, research, summarization, and KB maintenance with Qwen/Gemma-scale models.
  • Sophisticated multi-agent “harnesses” (Pi, Vibe, Claude Code-style setups, custom orchestrators) can extract more from smaller models but add complexity and security concerns.

Speech-to-Text and Ancillary Models

  • Whisper is no longer viewed as STT SOTA; Parakeet v3 and Voxstral cited as more accurate/faster.
  • Local STT + LLM assistants are considered practical on modest VRAM.

Security and Isolation

  • Strong advice to sandbox agents (VMs, microVMs, OS sandboxes, SELinux, restricted FS access).
  • GPU passthrough VMs are seen as trickier; some prefer LLM on host, agents in isolated guests.