Bonsai 27B: A 27B-Class model that runs on a phone

Quantization approach & “1-bit” details

• Thread clarifies that “1-bit” here often means ternary weights {−1,0,+1} or binary {−1,+1} with shared FP16 scales.
• Ternary variant: ~1.71 effective bits/weight using {−1,0,+1} + FP16 group scaling.
• Binary variant: ~1.125 effective bits/weight using {−1,+1} + scaling.
• Discussion on packing trits: common scheme is 5 trits in 8 bits (~1.6 bits/trit); more optimal packings exist but are less practical due to byte rounding and unpack cost.
• Some tooling currently uses 2-bit formats (Q2_0) that waste one code and duplicate scales; Prism’s fork partially fixes this via larger groups.

Tooling, deployment & hardware

• Models are on Hugging Face; many users report needing Prism’s llama.cpp fork or updated engines for compatibility.
• Locally AI app on iOS runs the 27B model but only on recent phones (e.g., iPhone 17 Pro/Pro Max) and requires app updates; some confusion over missing multimodal features from the demo.
• Reports of success on Mac (MLX, oMLX), some failures in LM Studio/Unsloth until updates land.
• CPU benchmarks: binary version reasonably fast; ternary CPU path currently much slower, suggesting unoptimized kernels.
• KV-cache usage is noted as unusually frugal, potentially valuable for multi-agent workflows.

Model quality, benchmarks & failure modes

• Several users find the ternary 27B “impressive” for deep learning/computer vision and coding questions, though general knowledge weaker.
• Others report reasoning “doom loops,” especially with high “thinking” settings, and a tendency to get stuck in repetitive tokens (e.g., “!!!!!!!”), which some link to Qwen-family quirks or extreme quantization.
• Tool-calling performance appears noticeably degraded vs higher-bit baselines; multi-turn benchmarks flagged as important to assess “doom loops.”
• Independent early lm-eval tests suggest Bonsai’s 4-bit variant may be over-quantized (huge perplexity jump), though there’s disagreement whether the eval was misconfigured.

Comparisons to other models

• Base is Qwen 3.6 27B; users contrast Bonsai to:
  • Qwen 3.6 27B full-precision / NVFP4 quants,
  • Qwen 35B MoE (often faster and less loopy),
  • Gemma 4 12B/31B, especially QAT 4-bit versions,
  • Other “edge” models (Ornith, Geryon, DeepSeek V4 Flash).
• Consensus: Bonsai looks very strong on math/coding for its size, weaker on tool use, knowledge breadth, and especially vision vs Gemma 4 12B QAT.

Use cases, limitations & demo critique

• Enthusiasm for fitting ~27B-class capability into ~3–4 GB and onto phones / 16 GB laptops; interest in scaling >100B params at ~1–2 bits.
• Some argue 27B is overkill for phones and that small, specialized models may be better than heavily quantized large models.
• Cooking demo is widely criticized: recipe quality is poor and highlights that generic “what to cook” queries are better handled by simple web search.
• Several users report hallucinations (e.g., misidentifying products) similar to other small/quantized models.

Business & ecosystem implications

• On-device models seen as a major shift for privacy-focused or “Privacy GPT”-style startups, and for regulated industries wanting self-hosted AI.
• Discussion notes that large vendors (phone makers, hyperscalers) are likely backing such work; efficient local models viewed as a strategic “open source GTM” and hardware differentiator.
• Some skepticism about marketing tone (“paradigm shift,” AI-generated-sounding blog posts), but broad agreement that 1–2 bit research is an important direction.