Running local models is good now

Original Article ↗ Hacker News Discussion ↗

Running modern AI models on personal hardware is becoming increasingly practical, with open‑weight options like Qwen and Gemma now fast and capable enough for many day‑to‑day tasks, especially on high‑RAM Macs or GPU workstations. Commenters weigh the trade‑offs between buying expensive local hardware and paying for frontier cloud models, noting that local setups offer privacy, control, and predictable behavior but still lag on complex, large‑scale coding and agentic workloads. Many expect a hybrid future where local models handle routine or privacy‑sensitive work while cloud models remain dominant for the most demanding use cases.

Hardware & affordability

  • Many comments say “local is good now” only if you have serious hardware: 24–32 GB VRAM GPUs, 64–128 GB RAM Macs, or Strix Halo / DGX Spark–class boxes.
  • Others push back: a 2–3k USD machine is out of reach for a large part of the world; for many devs $20–$200/month for a hosted model is cheaper and simpler.
  • There’s debate over whether laptops (especially thermally constrained ones) are a good idea vs desktop/workstation or small on‑prem servers.

Capabilities of current local models

  • Qwen 3.6 27B (dense) and 35B MoE, Gemma 4 (26B MoE, 31B dense), GLM 5.x, GPT‑OSS, Granite 4.1, etc. are cited as “very capable,” roughly comparable to mid‑2025 frontier quality for many tasks.
  • Users report 30–150+ tokens/s on commodity GPUs (e.g., 3090/4090/5090) and high‑RAM Macs, especially with MTP/speculative decoding and good quantization.
  • Diffusion-style LLMs (e.g., DiffusionGemma) impress some for single‑prompt speed, but researchers in the thread say they likely don’t scale or match dense transformers in quality.

Coding & agent workflows

  • Strong split in experience: some say Qwen 3.6 27B/35B are “good enough to daily‑drive” for coding; others find them far behind Claude/GPT for non‑toy codebases.
  • Local models often struggle with:
    • Reliable tool calls and JSON output.
    • Large context workflows (hundreds of k tokens).
    • Self‑directed, long‑horizon “vibe‑coding”; they get stuck, loop, or make subtle design errors.
  • Many succeed by:
    • Using frontier models to plan and local models to execute smaller tasks.
    • Keeping prompts extremely specific and scoping work tightly.
    • Accepting them as “smart autocomplete / junior dev,” not autonomous architects.

Harnesses, prompts & configuration

  • Repeated theme: harness matters as much as the model. Pi, OpenCode, Hermes, custom CLIs, and carefully tuned system prompts/AGENTS.md drastically change results.
  • Quantization is contentious: 4‑bit can “lobotomize” models for tool use; some advocate 5–6 bit or Q8 where possible, especially for MoE.
  • Many warn that local setups require significant tinkering with flags, quant schemes, context sizes, and templates; expectations should be set accordingly.

Cloud vs local economics & control

  • Pro‑local arguments:
    • No rate limits, rug‑pulls, silent “nerfs,” or model deprecations.
    • Better privacy and IP control; useful for regulated or paranoid environments.
    • Hardware may depreciate more slowly than people expect given current GPU scarcity.
  • Pro‑cloud arguments:
    • Frontier models (Claude/GPT/DeepSeek) still clearly smarter at complex coding and agentic work.
    • Hosted inference can be cheaper and faster for most users, with no hardware or maintenance burden.
    • Businesses often prefer to “outsource the headache,” even if on‑prem could be cheaper.

Future outlook

  • Many expect open/local models to keep improving with better architectures, quantization‑aware training, and inference tricks, but also expect frontier vendors to stay ahead at scale.
  • There’s interest in:
    • On‑prem GPU appliances for teams (e.g., “LLM in a closet”).
    • Hybrid patterns: local for routine/private/low‑latency work, cloud for heavy planning or huge contexts.
  • Some see this as a pivotal moment: local is already “good enough” for a surprising share of everyday tasks, though not yet a full replacement for top cloud models.