Exo: Run your own AI cluster at home with everyday devices

Original Article ↗ Hacker News Discussion ↗

A new open-source project, Exo, aims to let people pool everyday devices into a local cluster capable of running larger AI models, challenging the assumption that useful AI will remain cloud-only. Commenters weigh the appeal of local inference for privacy, censorship resistance, cost control, and experimentation against the superior quality and convenience of hosted models, while probing technical limits such as network latency, bandwidth, and hardware support. Many see the project as early and rough but promising, particularly if it can make multi-device AI as simple to run as consumer apps like Dropbox.

Motivations for running models locally

  • Privacy and censorship resistance are recurring reasons: users want to run on sensitive data (journals, private audio, “spicy” images) without sending it to large providers.
  • Customization is easier locally (changing system prompts, using uncensored models, LoRAs, domain-specific setups).
  • Offline and reliable access is valued, especially where connectivity is unreliable or providers could change policy or shut down.

Arguments for cloud-hosted models

  • Many note a large quality gap: small local models (e.g., 7–8B) are seen as far behind GPT‑4/Claude-level systems for complex or high-stakes work.
  • For productivity, $20–100/month in API usage is argued to be cheaper than buying and operating powerful local hardware, especially once you factor in setup and maintenance.
  • Hosted solutions offer integrations (web search, tools like Wolfram Alpha) that local models typically lack.

Cost and hardware trade-offs

  • One side: spare hardware + free open models = $0 experimentation; good for students and hobbyists. Cloud is “not free” and can get expensive for heavy use.
  • Other side: upfront cost of capable GPUs, electricity, and time is high; for “just messing around,” cheap APIs and free tiers (TogetherAI, Groq, OpenRouter) are seen as better.
  • Some argue that for sustained >8h/day workloads, owned or colo hardware can beat big-cloud pricing; others counter that cloud still benefits from economies of scale.

How Exo works and technical feasibility

  • Exo uses pipeline parallelism: different devices hold different layers; only activations (embeddings) are sent between them.
  • Reported activation sizes: ~8–10 KB per token for 8B models, ~32 KB for 70B; expected to stay O(10–100 KB) even for much larger models.
  • On a local network, bandwidth is seen as fine; latency is the main bottleneck, especially over the internet, limiting SETI@home-style global clustering.
  • Some users report no speedup when using two MacBooks versus one, suggesting current implementation or scheduling needs work.

Maturity, platform support, and concerns

  • Project is explicitly experimental and rapidly changing; issues include:
    • Early hard dependency on Apple-only MLX, conflicting with “everyday devices” marketing.
    • Tinygrad backend exists; llama.cpp support is planned.
    • Windows, iOS, Android, Raspberry Pi, and Coral TPU support are desired but not all are proven.
    • Lack of benchmarks (tokens/sec, latency) and initial missing license; both requested by users.
    • Security model currently assumes a trusted local network; documentation is being updated.

Broader themes

  • Debate over whether “swarm compute” of idle devices is desirable versus preserving device longevity, power, and thermals.
  • Some view local/self-hosted AI as philosophically similar to open source and as a check on concentrated corporate control.