Towards 1-bit Machine Learning Models

Original Article ↗ Hacker News Discussion ↗

Researchers and practitioners are increasingly experimenting with extreme quantization of neural networks, pushing large language models down to 1–2 bits per weight while trying to retain near–full precision accuracy. Commenters explore how these schemes work in practice – e.g., using binary/ternary matmuls, per-group scaling metadata offloaded to CPU RAM, and post-quantization fine-tuning with LoRA – and why they’re especially attractive for very large models where memory bandwidth and energy dominate costs. The conversation also highlights open questions around training at 1 bit, the limits of current GPU hardware, and the potential for future custom accelerators that could make such ultra-compact models vastly more efficient.

Low-bit and Sub-1-bit Models

  • Commenters note that 1‑bit and even “sub‑bit” effective precision has been explored for years (binary/ternary nets, noisy weights giving ~0.68 effective bits, BitNet b1.58, etc.).
  • There’s excitement about 1‑bit or ~1.5‑bit models for LLMs because parameter storage and compute could, in principle, drop by large factors while preserving accuracy.
  • Some speculate that model size (more parameters at fewer bits) could more than compensate for lower per‑weight precision.

Why 1-bit/ternary Can Work

  • Several comments connect this to information theory: natural language and learned representations are highly redundant, so many full‑precision weights are effectively noise.
  • Intuition: the main information in a weight is often just “is there a connection?” and “is it excitatory vs inhibitory,” which binary or ternary values can encode.
  • Ternary {-1,0,1} is framed as encoding inverse correlation / no correlation / correlation.
  • Some liken 1‑bit networks to logic circuits or decision trees built from simple gates.

Training with Discrete Weights

  • A recurring question: how do you do gradient descent with 1‑bit parameters?
  • Answers: keep high‑precision “latent” weights, optimizer state, and gradients; quantize to 1–2 bits only for forward (and sometimes backward) passes.
  • Purely discrete optimization (evolutionary methods) is discussed but viewed as far less efficient than gradient‑based methods, though a few speculate specialized hardware might change the trade‑offs.

Quantization Schemes and HQQ Details

  • The HQQ approach: quantize pretrained FP16 models to 1–2 bits, then recover quality by fine‑tuning small low‑rank adapters (LoRA).
  • Quantization uses per‑group scales and zero‑points (metadata). At 1‑bit, this makes the true effective bits per parameter >1, but metadata can be offloaded to CPU.
  • There is back‑and‑forth on whether reporting only GPU memory (excluding CPU‑resident metadata) is misleading; the HQQ author clarifies that at runtime only one layer’s metadata is on GPU, so extra VRAM is a few MB.

Performance, Hardware, and Kernels

  • Benchmarks in the post show 1‑ and 2‑bit HQQ models slower than FP16 on current GPUs because matmul is still done in FP16 via cuBLAS; only dequantization is custom CUDA.
  • In contrast, BitNet reports latency gains using custom low‑bit kernels.
  • Many note that current CPUs/GPUs aren’t optimized for 1‑bit linear algebra, despite having popcount and even some 1‑bit tensor core support; hand‑tuned kernels dominate real performance.
  • Several expect future ASICs or FPGA designs focused on XNOR/popcount and binary/ternary matmul to yield enormous speed and energy gains, especially if paired with in‑memory compute.

Architectures and Applicability

  • Commenters stress that 1‑/2‑bit methods are not LLM‑specific; binary/low‑bit nets have long been used in computer vision and generic MLPs.
  • Transformers are described as stacks of mostly linear layers plus attention, arguably simpler for quantization than many older architectures.
  • Some ask whether low‑bit models can also reduce depth/width if precision is increased, but others note residual layers already help the model learn an effective depth.

Speculation and Open Questions

  • There is discussion of “sub‑1‑bit graphs” as compressed adjacency matrices with structured sparsity.
  • People toy with “0.5‑bit,” negative‑bit, ternary, and quaternary schemes (e.g., {-1,0,1,2}) and links to radix economy, while others point out that choice of numeric base is largely orthogonal to quantization.
  • A few propose radically different hardware paradigms (large 2D grids of tiny LUTs, mesh‑connected cores, direct electronic/photonic circuit training).
  • Overall sentiment: strong enthusiasm about 1–2‑bit inference, cautious skepticism about fully 1‑bit training and real‑world hardware/practicality.