Show HN: Speeding up LLM inference 2x times (possibly)

Original Article ↗ Hacker News Discussion ↗

Overview of “Effort” / bucket-based dynamic sparsity

  • Technique replaces standard dense matvecs with a preprocessed, bucketed format where weights in each row are sorted by magnitude.
  • At inference, it dynamically selects only the most “important” weights per layer/token based on the current input vector and a global “effort” parameter (fraction of work to do).
  • Claims: around 2× faster matrix multiplications at ~25% effort on Apple Silicon, with outputs still close to baseline on informal tests. All weights remain available; they’re just not used all the time.

Relation to existing sparsity / pruning work

  • Compared to approaches like PowerInfer and MoE:
    • This works at single-weight granularity instead of whole neurons or experts.
    • It’s dynamic (input‑dependent) rather than statically pruning weights.
  • Commenters link it to magnitude pruning, SparseGPT, semi-structured (2:4) sparsity, mixture-of-depths, and low-rank ideas; author notes overlap but emphasizes runtime dynamic selection and a custom GPU-friendly matmul.

Implementation details & current limitations

  • Implemented in Swift/Metal targeting Apple GPUs; CPU speedups expected but not yet well-characterized.
  • Performance benefit is largely from reduced memory traffic; several commenters note speedups may diminish at larger batch sizes.
  • Current implementation has bugs (e.g., broken Mixtral path, suboptimal 100% mode), missing or initially misconfigured model artifacts, and documentation typos; work is in progress.
  • Q8 quantization support is incomplete; combining Effort with low-bit quantization is a major open question.

Quality, evaluation, and skepticism

  • Evaluation so far is mostly anecdotal (prompt comparisons, cosine similarity of activations). More rigorous benchmarks (e.g., KL-divergence, standard LLM evals) are requested.
  • One commenter characterizes the headline as over-strong given early, self-evaluated results; others echo “extraordinary claims require extraordinary evidence.”
  • Author repeatedly hedges with “possibly,” acknowledges unknowns (especially at Q8, smaller models, batch>1), and invites community validation.

Potential applications and extensions

  • Interest in integrating into llama.cpp, mobile and low-VRAM setups, MoE models, CNNs/diffusion, and cross-platform libraries.
  • Ideas raised: a small controller model to choose effort per layer/token, post-training or quantization-aware training to recover quality, and tuning effort unevenly across layers or MoE experts.