LLaMA now goes faster on CPUs

Original Article ↗ Hacker News Discussion ↗

Optimizations to matrix multiplication in the open-source LLaMA/llamafile stack are making large language models run much faster on commodity CPUs, narrowing (but not closing) the gap with GPU-based inference and enabling useful local models even on modest hardware. Commenters debate how meaningful this is in practice—touching on hardware trade‑offs, memory bandwidth limits, and the role of quantization—while also exploring broader questions about what LLMs actually capture (facts vs. tacit skills), how they should be used as “compressed knowledge” or offline backups, and why local, uncensored models matter for privacy and experimentation.

Local LLMs as “Knowledge Backups”

  • Some see local LLMs as a hedge against “Library of Alexandria”–style data loss: a compact, queryable snapshot of much of the public internet.
  • Others argue this is misguided:
    • LLMs are lossy, fuzzy “compressed” knowledge, not precise archives.
    • Traditional archives (Wikipedia dumps, Kiwix ZIM files, Project Gutenberg, etc.) are more reliable, smaller, and require far less compute.
  • Several note that much crucial human/tacit knowledge (craft skills, social know‑how, sensory experience) isn’t well captured in text or datasets anyway.

Capabilities and Limits of LLMs

  • Strong disagreement over whether current models “reason” or simply remix training data.
  • Examples of clever analogies and broad knowledge impress some; others see hallucinations and factual errors as disqualifying for serious reliance.
  • Small/local models are often weak compared to hosted ones, but interacting with them is praised as a way to understand failure modes and maintain healthy skepticism.
  • Many emphasize that models lack real-world grounding (object permanence, causality, sensations), and that future AGI likely requires online learning, agency, and new architectures.

Running Models Locally: Tools and UX

  • Recommended tools include llamafile, llama.cpp, Ollama, LM Studio, Msty, PrivateGPT, Koboldcpp, and various uncensored model variants.
  • Macs with M‑series chips and 30–70B‑class models are seen as a sweet spot; Raspberry Pi and older hardware can run tiny models but are slow and bandwidth-limited.
  • Local use is valued for privacy, avoidance of cloud subpoenas, and freedom from SaaS guardrails.

CPU vs GPU Performance and Matmul Work

  • The linked work implements highly tuned CPU matmul / matvec kernels, yielding big speedups over naïve or older code paths in llama.cpp, especially with AVX‑512, bfloat16, and cache-aware blocking.
  • Commenters stress that GPUs (and ASICs/TPUs) still dominate for throughput and energy efficiency, though fast CPUs plus better memory (e.g., MCR DIMMs) may narrow the gap for inference.
  • Discussion touches on BLAS libraries, compiler limits, loop unrolling, cache behavior, and the difficulty of matching vendor libraries like cuBLAS or MKL.

Safety, Censorship, and Uncensored Models

  • Hosted models’ safety filters (on “hacking,” self-harm, sexual violence, etc.) frustrate users doing legitimate research or creative writing.
  • Uncensored local models (including vision models) are highlighted as alternatives, though people warn that hallucinated instructions (e.g., in chemistry) can be dangerous.

Trust and Prior Claims

  • Some praise the optimization work and past contributions; others recall earlier overoptimistic claims (e.g., about mmap memory use) and call for independent benchmarking and cautious interpretation of speedup numbers.