Towards 1-bit Machine Learning Models

Original Article ↗ Hacker News Discussion ↗

Low-bit and Sub-1-bit Models

  • Commenters note that 1‑bit and even “sub‑bit” effective precision has been explored for years (binary/ternary nets, noisy weights giving ~0.68 effective bits, BitNet b1.58, etc.).
  • There’s excitement about 1‑bit or ~1.5‑bit models for LLMs because parameter storage and compute could, in principle, drop by large factors while preserving accuracy.
  • Some speculate that model size (more parameters at fewer bits) could more than compensate for lower per‑weight precision.

Why 1-bit/ternary Can Work

  • Several comments connect this to information theory: natural language and learned representations are highly redundant, so many full‑precision weights are effectively noise.
  • Intuition: the main information in a weight is often just “is there a connection?” and “is it excitatory vs inhibitory,” which binary or ternary values can encode.
  • Ternary {-1,0,1} is framed as encoding inverse correlation / no correlation / correlation.
  • Some liken 1‑bit networks to logic circuits or decision trees built from simple gates.

Training with Discrete Weights

  • A recurring question: how do you do gradient descent with 1‑bit parameters?
  • Answers: keep high‑precision “latent” weights, optimizer state, and gradients; quantize to 1–2 bits only for forward (and sometimes backward) passes.
  • Purely discrete optimization (evolutionary methods) is discussed but viewed as far less efficient than gradient‑based methods, though a few speculate specialized hardware might change the trade‑offs.

Quantization Schemes and HQQ Details

  • The HQQ approach: quantize pretrained FP16 models to 1–2 bits, then recover quality by fine‑tuning small low‑rank adapters (LoRA).
  • Quantization uses per‑group scales and zero‑points (metadata). At 1‑bit, this makes the true effective bits per parameter >1, but metadata can be offloaded to CPU.
  • There is back‑and‑forth on whether reporting only GPU memory (excluding CPU‑resident metadata) is misleading; the HQQ author clarifies that at runtime only one layer’s metadata is on GPU, so extra VRAM is a few MB.

Performance, Hardware, and Kernels

  • Benchmarks in the post show 1‑ and 2‑bit HQQ models slower than FP16 on current GPUs because matmul is still done in FP16 via cuBLAS; only dequantization is custom CUDA.
  • In contrast, BitNet reports latency gains using custom low‑bit kernels.
  • Many note that current CPUs/GPUs aren’t optimized for 1‑bit linear algebra, despite having popcount and even some 1‑bit tensor core support; hand‑tuned kernels dominate real performance.
  • Several expect future ASICs or FPGA designs focused on XNOR/popcount and binary/ternary matmul to yield enormous speed and energy gains, especially if paired with in‑memory compute.

Architectures and Applicability

  • Commenters stress that 1‑/2‑bit methods are not LLM‑specific; binary/low‑bit nets have long been used in computer vision and generic MLPs.
  • Transformers are described as stacks of mostly linear layers plus attention, arguably simpler for quantization than many older architectures.
  • Some ask whether low‑bit models can also reduce depth/width if precision is increased, but others note residual layers already help the model learn an effective depth.

Speculation and Open Questions

  • There is discussion of “sub‑1‑bit graphs” as compressed adjacency matrices with structured sparsity.
  • People toy with “0.5‑bit,” negative‑bit, ternary, and quaternary schemes (e.g., {-1,0,1,2}) and links to radix economy, while others point out that choice of numeric base is largely orthogonal to quantization.
  • A few propose radically different hardware paradigms (large 2D grids of tiny LUTs, mesh‑connected cores, direct electronic/photonic circuit training).
  • Overall sentiment: strong enthusiasm about 1–2‑bit inference, cautious skepticism about fully 1‑bit training and real‑world hardware/practicality.