I'm making LLMs work on small and old devices

Efforts to run increasingly large language models on surprisingly weak and aging hardware—from first‑gen iPads and TV boxes to routers, game consoles, and even vapes—are evolving into a kind of “can it run Qwen?” engineering challenge. Participants trade techniques like weight streaming, extreme quantization, and custom kernels, while debating whether tiny or highly optimized models can be practically useful for narrow, on‑device tasks such as transcription, control systems, or regulated, offline workloads. Many see this line of work as both a demoscene‑style hobby and a strategic push toward affordable, private AI that can live on ubiquitous low‑end devices rather than centralized data centers.

Project goals & current results

  • Project aims to run relatively large LLMs on very constrained or old hardware (sub‑1 GB RAM, no matmul/SIMD).
  • Reported: 16B models running on devices like an iPad Air 1st gen and Xiaomi TV box with “decent” speed.
  • New experiments claim 35B and even 80B models on phones, with a few tokens/sec and a target of ~10 tok/s via further optimization.
  • Models and kernels are being open sourced; modified models hosted on Hugging Face.

Suggested target devices & “Can it run Qwen?”

  • Many proposals: vapes, Playdate handheld, Raspberry Pi Zero/1/2/3/4, ESP32/ESP P4/S31, old iPhones, thin clients, Dreamcast, Wii, PS2 dev kits, routers (e.g., WRT54G), printers, SGI/Sun boxes, 80386 PCs, FPGA boards.
  • Spirit likened to the classic “Can it run Doom?”; some suggest “Can it run Qwen?” as the new meme.
  • Some community members already report LLMs on Wii, PS2, FPGA, Dreamcast, old iBooks, and thin clients.

Techniques & performance tricks

  • Heavy focus on custom kernels, streaming weights from flash/disk/optical media, extreme quantization, and per‑layer/partial loading approaches (e.g., per-layer embeddings, MoE-like routing).
  • FPGA work shows on‑chip or DDR‑bound LLMs; discussion of hitting DDR bandwidth limits.
  • Emphasis on automated “brute force” search for optimizations, often guided by larger LLMs (e.g., for code generation).

Usefulness of tiny models

  • Skeptical view: many tiny models are currently “too dumb” for general tasks; often only novelty, especially without fine‑tuning.
  • Counterpoint: fine‑tuned small models (1–27B) can perform very well on narrow domains (finance, legal, industrial control) and may outperform larger models on cost, deployment, and regulation.
  • Mention of tiny conversational/reasoning models (~6.5M params, ~30MB) with tool use and internal scripting via clever harnesses.
  • Other cited uses: speech recognition/generation on-device, log parsing, regulated on‑prem healthcare/HIPAA settings, edge IoT, standalone HVAC/industrial controllers.

Motivations & philosophy

  • Motivations include: hardware frugality, reusing e‑waste, privacy/air‑gapped deployment, enabling people with limited resources, and resisting a future where only big companies run huge models.
  • Some see this as “engineering fun” or demoscene‑like; others as groundwork for distributed “AI societies” of many small agents (“AI ants/bacteria/bees”).
  • Brief side debates about science education, corporate incentives, and potential social impacts; views are mixed and sometimes contentious.