I made an app that runs Mistral 7B 0.2 LLM locally on iPhone Pros

Original Article ↗ Hacker News Discussion ↗

Running large language models like Mistral 7B directly on iPhone Pro devices is now viable but still constrained by RAM, quantization choices, and thermal limits. Commenters compare multiple iOS and desktop apps for local inference, weighing factors such as model variety, speed (tokens per second), conversation history, and the trade‑offs between on‑device privacy and the superior quality of cloud models like GPT‑4. They also examine Apple’s hardware and software stack (Metal vs. Neural Engine), app-store policies, pricing fairness, and the likelihood that future apps will ship specialized, embedded models for offline use.

App capabilities & UX

  • App runs quantized Mistral 7B (and other small models) fully offline on recent iPhone Pros and some iPads/Macs.
  • Conversation history is a major UX concern: some tools lack it, while this app and others support saving, searching, resuming, and exporting chats.
  • Users want visibility into performance (tokens/sec, RAM use) and finer controls (model choice, temperature, system prompts, themes, haptics).

Performance, models, and device constraints

  • 7B models on phones are borderline: several reports of 7B at high-quality quantization being extremely slow (seconds per token) or not viable except on 16GB iPads/Apple Silicon Macs.
  • Lower-bit and smaller models (e.g., 1–3B, Q2–Q4 quantization) generally run acceptably; 7B often requires reduced context length.
  • Some note that datacenter GPUs can deliver >100 tokens/sec for quantized 7B models at batch size 1, far beyond phones.

Stability and crashes

  • Multiple users report hard device lockups, app hangs, and crashes (sometimes requiring forced reboot) when loading large or aggressively quantized models with Metal.
  • Developer responses describe new builds with:
    • Safer RAM headroom checks.
    • Fallback from GPU (Metal) to CPU when memory is tight.
    • Changing default models to lighter quantizations to reduce crashes.

System prompts & customization

  • Several complain that many iOS LLM apps don’t let users set system prompts, calling them “useless” without that.
  • Some alternative apps explicitly support editable system prompts and custom model presets.
  • There is debate over how Mistral’s chat template represents system prompts vs normal messages.

On-device vs server LLMs

  • Consensus: local models are slower and weaker than top hosted models (e.g., GPT-4), but attractive for privacy, offline use, experimentation, and niche workflows (RAG over personal data, function calling).
  • Discussion on throughput vs latency: batching improves server throughput but only modestly harms per-user speed if tuned well.

Apple hardware, ANE, and tooling

  • Most iOS LLM apps use CPU/GPU via Metal, not the Neural Engine, due to:
    • Lack of direct ANE APIs (CoreML only).
    • CoreML’s current limitations for LLMs (fixed shapes, weak quantization support, large memory footprints).
  • Some hope future Apple tooling (CoreML, MLX) will make ANE-based LLMs practical.

Pricing, reuse, and App Store issues

  • Price complaints center on paying for an app that doesn’t work on many devices; others respond that requirements are clearly stated and refunds exist.
  • One commenter alleges the app is largely a reskinned version of an existing free app.
  • Apple reportedly doesn’t enforce LLM content “safety” beyond age ratings; offensive outputs are not a blocker by themselves.