Exponentially faster language modelling

Original Article ↗ Hacker News Discussion ↗

Researchers present UltraFastBERT, a variant of the BERT language model that activates only about 0.3% of its neurons per inference using a tree-like “fast feedforward” (FFF) mechanism, yet reportedly matches standard BERT accuracy. Commenters highlight that this yields large speedups—tens to over 70× in their benchmarks—especially on CPUs and potentially enables much larger or cheaper models, faster edge inference, and new applications like on-device NPC logic or just‑in‑time embeddings. However, the method currently optimizes only inference (not training), still requires the full model in memory, and may face challenges scaling to very large transformer models and GPU-centric workflows.

Technique and Claims

  • UltraFastBERT replaces standard feedforward (FF) layers with “fast feedforward” (FFF) trees that select a tiny subset of neurons per inference.
  • Reported: ~0.3% of neurons used per layer, with performance comparable to standard BERT variants.
  • Benchmarks claim ~78× CPU speedup and ~40× PyTorch speedup for FF layers, with theoretical gains up to ~341× at BERT-base scale.
  • Training remains dense; the branching is only exploited at inference. The method relies on differentiable “soft” branching that is hardened over training.

Hardware and Implementation

  • GPUs are described as inefficient for branching; CPUs benefit more from conditional execution and sparse access.
  • Some commenters note CUDA benchmarks in the paper also show large speedups, but overall GPU memory remains a bottleneck since attention layers are unchanged.
  • FLOPs are seen as increasingly cheap; memory capacity and bandwidth dominate for large models.
  • Efficient support would benefit from new hardware primitives for conditional neural execution and better caching.

Applicability and Use Cases

  • In principle, applicable to other transformers (e.g., LLaMA/Mistral), since it mainly swaps the FF submodule, but existing pretrained weights cannot be reused; models must be retrained.
  • Potential applications: faster CPU-based inference, edge/phone deployments, game NPCs, just‑in‑time embeddings for search, larger batch and tree-of-thought inference, and possible gains for text-to-image or vision transformers.
  • Some speculate about distributing training/inference across many weaker devices, even volunteer-compute style, though others call this speculative.

Limitations and Open Questions

  • For large LLMs, attention often dominates compute, so FF-only speedups may yield smaller overall gains.
  • Modern inference is often memory‑bandwidth‑bound; it is unclear how much this approach reduces effective bandwidth needs, since all weights still exist even if sparsely used.
  • Sparse activations may risk degraded precision/recall in some settings.
  • The practicality of GPU implementations with heavy branching and the real-world impact on end-to-end latency remain open.

Interpretability and Research Context

  • The tree-structured routing resembles decision trees or hierarchical softmax, potentially improving interpretability and enabling “surgical” model edits and partitioning of input space.
  • Discussion notes that impactful foundational work often comes from smaller research groups, with concerns about conference incentives and “pay to play” dynamics.
  • Some express broader unease about AI progress driven by profit motives; others argue the issue is general incentive alignment rather than capitalism per se.