LLaMA now goes faster on CPUs

Original Article ↗ Hacker News Discussion ↗

Local LLMs as “Knowledge Backups”

  • Some see local LLMs as a hedge against “Library of Alexandria”–style data loss: a compact, queryable snapshot of much of the public internet.
  • Others argue this is misguided:
    • LLMs are lossy, fuzzy “compressed” knowledge, not precise archives.
    • Traditional archives (Wikipedia dumps, Kiwix ZIM files, Project Gutenberg, etc.) are more reliable, smaller, and require far less compute.
  • Several note that much crucial human/tacit knowledge (craft skills, social know‑how, sensory experience) isn’t well captured in text or datasets anyway.

Capabilities and Limits of LLMs

  • Strong disagreement over whether current models “reason” or simply remix training data.
  • Examples of clever analogies and broad knowledge impress some; others see hallucinations and factual errors as disqualifying for serious reliance.
  • Small/local models are often weak compared to hosted ones, but interacting with them is praised as a way to understand failure modes and maintain healthy skepticism.
  • Many emphasize that models lack real-world grounding (object permanence, causality, sensations), and that future AGI likely requires online learning, agency, and new architectures.

Running Models Locally: Tools and UX

  • Recommended tools include llamafile, llama.cpp, Ollama, LM Studio, Msty, PrivateGPT, Koboldcpp, and various uncensored model variants.
  • Macs with M‑series chips and 30–70B‑class models are seen as a sweet spot; Raspberry Pi and older hardware can run tiny models but are slow and bandwidth-limited.
  • Local use is valued for privacy, avoidance of cloud subpoenas, and freedom from SaaS guardrails.

CPU vs GPU Performance and Matmul Work

  • The linked work implements highly tuned CPU matmul / matvec kernels, yielding big speedups over naïve or older code paths in llama.cpp, especially with AVX‑512, bfloat16, and cache-aware blocking.
  • Commenters stress that GPUs (and ASICs/TPUs) still dominate for throughput and energy efficiency, though fast CPUs plus better memory (e.g., MCR DIMMs) may narrow the gap for inference.
  • Discussion touches on BLAS libraries, compiler limits, loop unrolling, cache behavior, and the difficulty of matching vendor libraries like cuBLAS or MKL.

Safety, Censorship, and Uncensored Models

  • Hosted models’ safety filters (on “hacking,” self-harm, sexual violence, etc.) frustrate users doing legitimate research or creative writing.
  • Uncensored local models (including vision models) are highlighted as alternatives, though people warn that hallucinated instructions (e.g., in chemistry) can be dangerous.

Trust and Prior Claims

  • Some praise the optimization work and past contributions; others recall earlier overoptimistic claims (e.g., about mmap memory use) and call for independent benchmarking and cautious interpretation of speedup numbers.