Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

Original Article ↗ Hacker News Discussion ↗

A new “Bonsai 2” version of the Qwen 3.8 27B language model claims near‑lossless accuracy while compressing the weights into a 5.9 GB ternary format, small enough to run on 8–16 GB consumer GPUs, Apple Silicon, and even in a web browser. Early users report impressive throughput for short tasks and tight‑VRAM setups but mixed results on longer, agentic coding jobs, with frequent looping and weaker real‑world performance than benchmarks suggest. Much of the conversation focuses on how it compares to existing quantization schemes like Unsloth and ISTA, the practical limits on context length and VRAM, and whether such aggressive compression can really preserve the base model’s capabilities.

Hardware, VRAM, and Performance

  • Main appeal: 27B Qwen 3.8-derived model in ~5.9 GB, making 16 GB GPUs and midrange hardware viable.
  • Users report running it on:
    • NVIDIA: 3070 (8 GB, borderline), 3060 (12 GB), 4090, 5090, 6000 Pro Blackwell, older 16 GB cards.
    • AMD: 6700 XT, 7900 XT/XTX; inference works but requires Prism’s fork, HIP-specific tuning, and sometimes slower paths.
    • Apple Silicon: M1/M2/M4/M5 devices see ~7–40 tok/s, with Metal “tensor API” issues in the current fork.
    • WebGPU / browser: Runs in desktop browsers; fails or is unstable on some phones (e.g. Pixel 9).
  • 6 GB GPUs can load it but are very slow (~0.7 tok/s).
  • Effective weight size ~5.9 GB, but extra VRAM is needed for KV cache; 8 GB may work only with short contexts.

Setup, Tooling, and Compatibility

  • GGUFs require Prism’s llama.cpp fork; upstream llama.cpp does not yet support the ternary format.
  • Users share build flags, server commands, and quantization/cache settings to avoid OOM and improve speed.
  • No drafter/speculative-decoding model is available yet for Bonsai 2; Spark/DGX users see limited benefit from n‑gram speculation.

Quality, “Near-Lossless” Claims, and Benchmarks

  • Benchmarks on the HF card suggest performance close to strong 4‑bit quants, which some find “crazy good” if accurate.
  • Several users report:
    • Frequent looping, especially in WebGPU and longer tasks.
    • Noticeably worse long-horizon reasoning, agentic coding, and memorized text recall vs. full-precision Qwen 3.8 27B.
    • Good short free-form outputs but disappointing as a coding agent.
  • Some think benchmarks are cherry-picked (few long-context or multi-step tasks) and that “near-lossless” should be treated skeptically.

Comparisons to Other Quants and Models

  • Compared against Unsloth quants (UD-Q4, 2–3 bit variants), ISTA’s 3-bit Qwen quant, and other advanced schemes (e.g., AngelSlim).
  • Thread consensus: naive ≤4 bpw quants degrade quickly, but sophisticated QAT/PTQ (like Bonsai) can push below 2 bpw; implementation details are proprietary or complex.
  • Some users now favor alternative 3–4 bit GGUFs (Unsloth, ISTA) for quality, using Bonsai mainly when VRAM is the bottleneck.

Use Cases, Limitations, and Future Directions

  • Strong interest in:
    • Phone-friendly 8B-class Bonsai based on Qwen 3.8.
    • Very large compressed models (100B+ or Flash/Next variants) that might fit in high-end consumer VRAM.
  • Multiple reports that on real “agentic” coding tasks the model stalls, loops, or can’t decide on an approach, while cloud frontier models complete the same tasks reliably.

Language and Metrics Discussion

  • Lengthy side debate about phrases like “9x smaller”:
    • Some argue it’s mathematically nonsensical; others treat it as a widely understood idiom meaning “1/9 the size.”
    • Similar arguments appear around “N× faster” and unit choices like mWh/token.