llama.cpp

Original Article ↗ Hacker News Discussion ↗

Llama.cpp, a popular open-source C/C++ runtime for running large language models locally, is drawing renewed attention with its new llama.app website and curl‑based installer, prompting both enthusiasm and security concerns around ease of installation versus “curl | sh” risks. Commenters broadly praise llama.cpp’s performance, hardware support (NVIDIA, AMD, Intel, Vulkan, SYCL, ROCm), and flexibility—especially for multi-model setups and agentic coding workflows—while noting rough edges, regressions on some GPUs, and a learning curve for non-experts. There is ongoing comparison with higher-level tools like Ollama and LM Studio: many see those as friendlier wrappers that helped popularize local LLMs, but argue that llama.cpp ultimately offers better control, a richer GGUF ecosystem, and fewer vendor complications if users are willing to manage builds and configuration themselves.

Legitimacy and New Website

  • Some were initially suspicious of llama.app, but it’s linked from the official llama.cpp repo and has its own public site repo.
  • Site is seen as “vibe-coded” and marketing-heavy; some like the friendlier presentation, others find it amateurish or misleading for not prominently stating its origins or non‑Meta status.
  • One commenter worries about trademark issues with Meta’s LLaMA; others note llama.cpp has existed for years without apparent conflict.

Installation and curl|bash Debate

  • New one‑liner curl https://llama.app/install.sh | sh is a major discussion point.
  • Many dislike curl|sh on security and transparency grounds, preferring git clone + cmake, OS packages (Homebrew, Arch, etc.), or prebuilt binaries.
  • Counter‑arguments: installing from a trusted HTTPS source isn’t fundamentally different from other software installs; ease of installation is crucial for adoption.
  • Some highlight benefits of package managers (cryptographic attestation, predictable uninstall), and isolation via containers/VMs for agents and tools.

Backends, Hardware, and Performance

  • Experiences across Intel Arc, NVIDIA, AMD/ROCm, and Vulkan are mixed:
    • Building with OpenVINO/SYCL can be tricky for some Intel Arc users.
    • On AMD, ROCm support is fragile; several recommend just using Vulkan, or alternative engines like Hipfire or wrappers like Lemonade-server and vendor “toolboxes.”
  • Reports that vanilla llama.cpp leaves performance on the table, but others show significant gains from careful startup parameter tuning and speculative decoding.
  • On Apple Silicon, performance of MLX vs llama.cpp is now close; GGUF ecosystem and caching behavior are important factors.

llama.cpp vs Ollama and Other Runtimes

  • llama.app is seen as a direct Ollama competitor (CLI llama serve, one‑liner install, branding).
  • Strong disagreement over whether Ollama “uses” llama.cpp vs just ggml and its own kernels; some criticize Ollama for past crediting behavior.
  • Several say Ollama is easier and more popular, but llama.cpp is more flexible, faster on their hardware, and closer to the “real thing.”
  • LM Studio and Kobold are noted as GUIs/wrappers that internally rely on llama.cpp.

Usability, Stability, and Project Direction

  • Some praise llama.cpp as the “ffmpeg of AI”: fast to adopt new models, widely supported, and high quality.
  • Others complain about regressions (especially ROCm) and the “move fast, break things” feel on master; advice is to pin working commits.
  • There’s criticism that the project historically hasn’t been as “click‑and‑run” friendly as Ollama; defenders argue it’s server software and GUIs should handle UX.

Models, Use Cases, and Local Agents

  • Users run a variety of models (Qwen 3.x, Gemma 3, Llama 3.2, DeepSeek V4 Flash, etc.) in GGUF, often heavily quantized.
  • Some find small models (e.g., ~4–12B) limited for serious coding or reasoning, especially with short context windows; others report success with careful prompting.
  • Discussion around the cheapest viable local agentic coding setup: suggestions include used RTX 3090s or energy‑efficient Intel Arc builds; tradeoffs revolve around VRAM, memory bandwidth, and power.

Multi‑Model Routing and Tooling

  • llama-server now supports multi‑model loading and routing, with an evolving “router mode”; earlier solutions like llama-swap still offer richer routing and UIs but add complexity.
  • KV‑cache handling, speculative decoding, and batching are key to performance; some automate config tuning using an LLM itself.
  • Mention of harnesses built on llama-cpp-python and minimal agents directly on llama.cpp (e.g., DLLM in D) for efficient local coding workflows.