Ollama now supports AMD graphics cards

Original Article ↗ Hacker News Discussion ↗

Significance of AMD Support & Impact on Nvidia

  • Many see AMD support in Ollama as weakening Nvidia’s CUDA moat, at least on the inference side.
  • Training is still viewed as Nvidia-dominated, but inference is more about cost per token and VRAM, where AMD can compete.
  • Local LLM enthusiasm is pushing demand for non-Nvidia options due to price and availability.

Performance, Cost, and VRAM Tradeoffs

  • Multiple comments note AMD cards generally trail RTX 4090 in raw performance, but are cheaper per GB of VRAM.
  • For many users, being able to run larger/better models at slightly lower tokens-per-second is preferred over faster but VRAM-limited Nvidia cards.
  • Some report concrete numbers (e.g., ~35 tokens/s on Mistral 7B Q6 on a midrange AMD GPU), but systematic benchmarks are largely “unclear”.

Implementation Details: ROCm, Vulkan, OpenCL, HIP

  • Ollama’s AMD support is built on llama.cpp, using ROCm; a linked commit confirms ROCm integration.
  • Vulkan and OpenCL backends exist in llama.cpp for older AMD GPUs; performance is described as worse than CUDA/ROCm but usable.
  • HIP is mentioned as a CUDA-like portability layer for AMD, though reliability and stability are debated.
  • OpenCL is widely described as “alive but second-class” and less pleasant to use than vendor-specific stacks.

Hardware Compatibility & Workarounds

  • Official support lists focus on newer RDNA2/3 cards; many users report “unofficial” success on more GPUs via environment variables (e.g., HSA_OVERRIDE_GFX_VERSION=10.3.0).
  • Some cards (notably 5700XT) lack ROCm support, frustrating users who upgraded from older Nvidia cards.
  • Integrated Radeon/APU support is not yet available; issue links indicate it’s a known limitation.

Ollama vs. llama.cpp and Other Tools

  • Broad agreement that Ollama is essentially a user-friendly wrapper around llama.cpp, adding a registry, simple CLI, and automatic VRAM-based configuration.
  • Several users say they struggled to compile llama.cpp (especially with CUDA/ROCm) but found Ollama “one-command” easy.
  • Others prefer direct llama.cpp, koboldcpp, or other UIs for more control, performance tuning, concurrency, or multiple models.

Production vs. Experimentation Debate

  • Some argue Ollama is mainly for local experimentation and personal use, lacking concurrency and multi-model serving, so not ideal for enterprise/production.
  • Others report using Ollama in “production-like” personal or small-scale workflows (RAG, local knowledge extraction, on-prem tools) and value its simplicity.
  • There is concern that llama.cpp does most of the hard optimization work while Ollama captures much more mindshare, and discussion around whether Ollama sufficiently credits its dependency and complies with licensing.