Show HN: Speeding up LLM inference 2x times (possibly)

Original Article ↗ Hacker News Discussion ↗

A new open-source technique called “Effort” claims to accelerate large language model inference by dynamically skipping low-impact weight multiplications, achieving up to 2x speedups on matrix–vector operations while sometimes loading only a subset of model weights into memory. Commenters compare the method to existing sparsity, pruning, MoE, and quantization approaches, probing how it trades accuracy for speed, how well it might work on CPUs, GPUs, and mobile devices, and whether it stacks with 8-bit quantization. While early tests are promising and the idea is considered easy to experiment with on open models like Mistral and Mixtral, several people stress the need for more rigorous benchmarks, especially at lower precision and larger batch sizes.

Overview of “Effort” / bucket-based dynamic sparsity

  • Technique replaces standard dense matvecs with a preprocessed, bucketed format where weights in each row are sorted by magnitude.
  • At inference, it dynamically selects only the most “important” weights per layer/token based on the current input vector and a global “effort” parameter (fraction of work to do).
  • Claims: around 2× faster matrix multiplications at ~25% effort on Apple Silicon, with outputs still close to baseline on informal tests. All weights remain available; they’re just not used all the time.

Relation to existing sparsity / pruning work

  • Compared to approaches like PowerInfer and MoE:
    • This works at single-weight granularity instead of whole neurons or experts.
    • It’s dynamic (input‑dependent) rather than statically pruning weights.
  • Commenters link it to magnitude pruning, SparseGPT, semi-structured (2:4) sparsity, mixture-of-depths, and low-rank ideas; author notes overlap but emphasizes runtime dynamic selection and a custom GPU-friendly matmul.

Implementation details & current limitations

  • Implemented in Swift/Metal targeting Apple GPUs; CPU speedups expected but not yet well-characterized.
  • Performance benefit is largely from reduced memory traffic; several commenters note speedups may diminish at larger batch sizes.
  • Current implementation has bugs (e.g., broken Mixtral path, suboptimal 100% mode), missing or initially misconfigured model artifacts, and documentation typos; work is in progress.
  • Q8 quantization support is incomplete; combining Effort with low-bit quantization is a major open question.

Quality, evaluation, and skepticism

  • Evaluation so far is mostly anecdotal (prompt comparisons, cosine similarity of activations). More rigorous benchmarks (e.g., KL-divergence, standard LLM evals) are requested.
  • One commenter characterizes the headline as over-strong given early, self-evaluated results; others echo “extraordinary claims require extraordinary evidence.”
  • Author repeatedly hedges with “possibly,” acknowledges unknowns (especially at Q8, smaller models, batch>1), and invites community validation.

Potential applications and extensions

  • Interest in integrating into llama.cpp, mobile and low-VRAM setups, MoE models, CNNs/diffusion, and cross-platform libraries.
  • Ideas raised: a small controller model to choose effort per layer/token, post-training or quantization-aware training to recover quality, and tuning effort unevenly across layers or MoE experts.