Show HN: 1-Bit Bonsai, the First Commercially Viable 1-Bit LLMs

Original Article ↗ Hacker News Discussion ↗

Commercially viable 1‑bit large language models are starting to appear, with PrismML’s “Bonsai” offering Qwen‑based 4B–8B models squeezed into roughly 1–1.5 GB while running at very high token speeds on consumer and even mobile hardware. Commenters report behavior reminiscent of early GPT‑3—coherent on many coding, SQL, and writing tasks but prone to vivid hallucinations—and debate whether the real breakthrough is density and speed rather than raw capability. There is enthusiasm about using such models for constrained or tool-augmented workloads, alongside skepticism about benchmark comparisons to full‑precision models and open questions about training cost and how well this 1‑bit approach will scale to 30B–100B parameters.

Quantization Approach & “1‑Bit” Details

  • Weights are stored as 1‑bit values in groups of 128, each sharing a 16‑bit scaling factor; effective precision is ~1.1 bits, not pure 1‑bit.
  • Some compare this to earlier 1.58‑bit / ternary work and ask how it scales to larger models (27B, 35B, 100B+).
  • There’s interest in theoretical work on fully binary training and backprop, but Bonsai appears to be a quantized Qwen variant, not trained from scratch in binary.

Performance, Quality & Trade‑offs

  • Benchmarks in the whitepaper put the 8B model below larger mainstream models (e.g. Qwen3) in accuracy but at dramatically smaller size (16× smaller) and much faster inference (≈6× on an RTX 4090).
  • Users report:
    • Very fast generation (hundreds of tokens/s on high‑end GPUs, workable on older CPUs and phones).
    • Quality reminiscent of early GPT‑3: often coherent and useful for coding, SQL, LaTeX, simple data tasks; but frequent hallucinations and factual mistakes.
    • Fails some reasoning tests (e.g. “car wash” distance, strawberry test, timezone conversions), and produces nonsense in some factual domains (e.g. physics, Harry Potter lore).

Deployment Experiences

  • Runs via a fork of llama.cpp, with special kernels and a custom quantization type; building from source and checking out the right branch is required.
  • Some struggle with gibberish output until they use the correct fork/branch or parameters (e.g. context size, AVX2, KV cache precision).
  • Works on Jetson, older laptops, iPhones (via third‑party apps), and consumer GPUs; CPU‑only is possible but can be slow without optimizations.
  • Memory usage in practice sometimes closer to 4‑bit quants than the headline “14× less,” leading to confusion.

Use Cases & Outlook

  • Seen as promising for: lightweight agents, classification, translation, simple summarization, SQL agents, and as sub‑components under stronger “orchestrator” models.
  • Some expect future systems to rely more on small, tool‑using models rather than memorizing facts.
  • Enthusiasm about 1‑bit models as a path to democratized, large‑parameter local LLMs coexists with skepticism about missing comparisons against strong 4‑/8‑bit quantized baselines and unclear training cost.