How to setup a local coding agent on macOS

Original Article ↗ Hacker News Discussion ↗

Local coding agents on macOS are becoming practical for real work, but trade-offs between speed, model quality, and hardware requirements are central. Commenters compare tools like llama.cpp, oMLX, LM Studio, and Ollama, noting that direct llama.cpp setups and MoE models can significantly outperform higher-level wrappers, especially on high-RAM Apple Silicon machines. Many see local models as valuable for privacy, offline use, and learning how LLMs work, while acknowledging they remain slower and generally weaker than top hosted models, making them best suited for “autocomplete on steroids” and exploratory coding rather than full Claude/GPT-style automation.

Video and demo context

  • Some readers couldn’t see the embedded video; direct and full links were shared.
  • The demo is real-time and shows a “usable” speed, but one clip is cut before the model responds, so it’s not a perfect latency benchmark.

Tooling: llama.cpp vs Ollama vs LM Studio vs oMLX

  • Several argue you don’t need huggingface-cli; llama.cpp’s -hf (and -hfd for draft) can fetch models directly, with LLAMA_CACHE controlling storage.
  • Ollama is seen by some as convenient (GUI, easy integration with agents), by others as slower and unnecessary since it wraps llama.cpp and adds overhead; one link warns against using it.
  • LM Studio is praised for ease of use, good Metal performance, and a nice UI, but some prefer pure open-source stacks and direct CLI control.
  • oMLX is highlighted as making MLX-based servers easy, doing caching and model selection, and integrating with various agent harnesses and UIs; others report mixed performance vs GGUF/llama.cpp.

Performance, MTP/QAT, and model choices

  • Mixed experiences with MTP: sometimes improves time to first token, sometimes negligible or even harmful (e.g., markup issues, little gain on MoE models).
  • QAT and speculative decoding can make dense ~27–31B models more acceptable; MoE vs dense tradeoffs are discussed.
  • Benchmarks that use ~128 tokens are criticized as misleading; longer prompts and context lengths are needed for meaningful numbers.

Hardware and feasibility on Macs

  • Experiences range from 16GB Airs up to 128GB M4/M5 Max machines.
  • Consensus: large dense models need ≥48–64GB for comfort; smaller or more aggressively quantized models will run on 16–24GB but may be slow and CPU-bound.
  • Some consider local models “toys” versus cloud in both speed and capability; others find them fully “good enough” for offline coding and experimentation.

Quality vs speed and use cases

  • Several complain that articles focus on tokens/sec and ignore answer quality.
  • Others respond that base model quality is benchmarked elsewhere; the article is about running the model, so speed is the main variable.
  • General agreement that local models are more like “super autocomplete” than full autonomous agents, but useful for boilerplate, explanation, offline help, and privacy-sensitive work.

Local vs cloud, cost, and philosophy

  • Economic skeptics argue extra hardware cost rarely beats simply paying for cloud APIs.
  • Enthusiasts value privacy, offline reliability, learning how systems work, avoiding cloud dependence, and being able to swap models/agent harnesses.
  • There’s tension between “let AI be your subordinate to get work done” and concerns about over-reliance and “slop” production.