I want everything local – Building my offline AI workspace

Original Article ↗ Hacker News Discussion ↗

Motivations for Local/Offline AI

  • Strong desire to keep data on-device: avoid cloud training on private code, documents, or chats; distrust of AI companies’ actual practices vs. their T&Cs.
  • Some argue local setups meaningfully improve privacy, especially when air-gapped at runtime; others note OS, GPU drivers, package managers, and GUIs still send telemetry during install/use.
  • Skepticism that users will punish cloud providers for privacy violations, given continued usage of Google, Meta, OpenAI.

Ease of Setup vs. Real-World Accessibility

  • Some claim running a basic local LLM (e.g., via Ollama) is just a few commands; others push back that this is “easy” only for a small, technical minority.
  • Friction appears higher when adding sandboxed code execution (Apple containers, Docker) and tool-calling/agent workflows.

Hardware, Performance, and Quality Gap

  • Many see hardware as the main bottleneck: good local LLMs often need high-end GPUs or large unified memory (e.g., expensive Macs, Strix Halo boxes).
  • Debate over economics: one side sees local hardware as a rapidly depreciating, power-hungry hobby vs. cheap API access; the other points out that a few months of cloud GPU or Bedrock usage can already exceed a modest homelab’s purchase price.
  • Widely acknowledged gap between top local/open models and frontier cloud models (Claude, GPT-5), especially for coding and tool use. Benchmarks are seen as misleading vs. “feel” in real workflows.
  • Some report good experiences with mid-sized local models (e.g., ~20–30B) on Apple Silicon or consumer GPUs; others find speeds and tool-calling reliability unacceptable for serious work.

Use Cases, Tooling, and Stacks

  • Various stacks discussed: Ollama, Open WebUI, LM Studio, coderunner (Apple containers), Dockerized alternatives, BrowserOS, Kasm, MLX, etc.
  • A major missing piece for local coding assistants is robust tool calling and file access; many “tool-capable” models still say they can’t read files or hallucinate tool outputs.
  • RAG/knowledge layer is highlighted as the companion challenge: indexing personal emails, code, and documents can balloon vector DBs to tens or hundreds of GB. LEANN is discussed as a storage-efficient alternative.

Philosophy and Future Outlook

  • Some see local AI as mostly hobbyist today; necessary mainly for privacy, regulation (GDPR), or SME deployments where cloud is forbidden.
  • Others frame it as the new FLOSS-style movement: critical for retaining technical sovereignty and avoiding future pricing shocks and vendor “rug pulls.”
  • Expectations differ: some think open/local models will approach “good enough” parity; others believe SOTA will always stay materially ahead in the cloud.