Running local models on an M4 with 24GB memory

Original Article ↗ Hacker News Discussion ↗

Running modern open-weight LLMs on Apple’s M-series Macs is now practical but comes with sharp tradeoffs in RAM requirements, speed, and capability. Commenters report that 24–64 GB machines can handle 9–35B‑parameter models like Qwen and Gemma for tasks such as basic coding, file handling, and office automation, but that these setups remain noticeably weaker and slower than frontier cloud models for complex reasoning and large codebases. The main reasons to invest in high‑RAM local hardware are offline use, privacy, and predictable costs, while most agree that cloud subscriptions still offer better raw performance and convenience for intensive or revenue-generating work.

Hardware & Config Debates

  • Initial confusion over “M4 with 24GB” is resolved: it refers to Apple M4 Macs (Air/Pro/Mini), not Nvidia Tesla GPUs.
  • People share configs from 16GB Airs up to 128GB M5 Max MacBook Pros and 128GB desktops with GPUs.
  • Strong sentiment that RAM capacity often matters more than raw CPU/GPU for local LLMs; 32–64GB is “usable,” 96–128GB considered a sweet spot for serious work.
  • Some argue high-end Macs are poor value vs. cheaper desktops with used GPUs and lots of RAM/VRAM; others prefer paying once for a powerful laptop over ongoing cloud fees.

Model Choices & Performance

  • Qwen 3.6/3.7 and Gemma 4 are repeatedly cited as current “good enough” local models, especially 27B–35B variants; 9B models are often described as weak for serious coding.
  • 4–14B models are said to fall between GPT‑3.5 and GPT‑4o‑mini; still notably behind current frontier models.
  • Reported speeds on Apple silicon for 20–31B models cluster around ~7–12 tokens/s with 8‑bit or Q4/Q5 quants; MoE models can have decent tokens/s but poor time‑to‑first‑token.
  • Benchmarks and anecdotes show Gemma 4 31B and Qwen 3.6 27B/35B can sometimes rival older frontier behavior on constrained tasks, but not consistently.

Local vs Cloud Tradeoffs

  • Several participants stress that local models are “nowhere near” Claude Opus / ChatGPT‑5.x for complex coding, long‑context reasoning, and reliability.
  • Others report local models solving nontrivial tasks (debugging, protocol reverse‑engineering, security analysis) and being “good enough” for much daily work.
  • Economic arguments: a multi‑thousand‑dollar laptop vs. decades of a $20/month subscription; local only makes sense to some if offline use, privacy, or latency are critical.

Use Cases & Workflows

  • Effective use often involves interactive, step‑by‑step workflows, tight prompts, and frequent testing rather than long autonomous runs.
  • Local models are seen as strong for boilerplate coding, small refactors, office drudgery (email, translation, simple docs); weaker for large projects and high‑risk legal/finance tasks.
  • Some propose hybrid flows: frontier models for research/planning, local models for execution and editing.

Optimizations, Tooling & Meta

  • New inference tricks (MTP, turboquant, Dflash, rotorquant) and engines (mlx, llama.cpp, LM Studio, Ollama, browser‑based agents) are actively explored; people believe speed headroom remains.
  • There’s visible “bipolar” sentiment: excitement about technical progress and decentralization, alongside concern about overhyping local models and the impact of LLMs on software craftsmanship.