Brave Leo now uses Mixtral 8x7B as default

Original Article ↗ Hacker News Discussion ↗

Brave has switched its built‑in AI assistant Leo to the Mixtral 8x7B model, prompting comparisons with GPT‑4 and detailed talk about how to run Mixtral locally using tools like llama.cpp, koboldcpp, and ollama on consumer CPUs, GPUs, and Apple Silicon. Commenters weigh trade‑offs between quantization levels, memory requirements, and token speed, and also catalog cloud providers serving Mixtral (e.g., Mistral’s own API, Together, Groq, OpenRouter, Anyscale). Alongside the model choice, Brave’s privacy claims, adblocking approach under Chrome’s Manifest V3 changes, and features like unlinkable subscription tokens and omnibar-based queries draw both praise and skepticism over trust and long‑term viability.

Running Mixtral Locally

  • Users report successful local runs via llama.cpp, koboldcpp, text-generation-webui, Oobabooga, LM Studio, Ollama, and Apple Silicon–optimized builds.
  • Memory needs are high: even 2-bit quantizations require ~20GB RAM; 4-bit can barely fit in 32GB; 6–8 bit generally need 64GB+.
  • Offloading layers to GPU (via flags like --n-gpu-layers) yields big speedups until VRAM is near full; beyond that, offloading can become slower than CPU-only.
  • Apple Silicon with large unified RAM (e.g., 36–64GB) is repeatedly cited as “surprisingly fast” and convenient.
  • AMD/ROCm and integrated GPUs can work but setup is more complex; performance on integrated graphics is described as “dead slow” by some.
  • Jetson boards are debated: some say they’re fun and capable; others call the software stack fragile and unsuitable as a general LLM box.

Quantization & Precision

  • Heavier quantization (2-bit) is widely called “terrible” for serious use; 4-bit+ is seen as a better tradeoff.
  • Several explanations: neural nets are noise-tolerant; much of 16/32-bit float precision is redundant; modern quantization uses block-wise lookup tables and extra scale metadata.
  • Distillation and dropped weights are mentioned as alternative compression techniques.

Remote Mixtral APIs and Performance

  • Multiple providers are named: OpenRouter, Together, Fireworks, Anyscale, Mistral’s own API, Replicate, and Groq.
  • Together and Groq are praised for speed; Groq claims extremely high tokens/s for Mixtral.
  • Some use Mistral or Anyscale for production and small local quantized models for experimentation.

Brave, Manifest V3, and Adblocking

  • Users like Brave’s built-in adblocker and Chromium performance, and some see uBlock Origin adding little on top.
  • Debate over Manifest V3:
    • One side says MV3 still allows powerful adblocking with large rule limits and dynamic rule updates.
    • The other argues MV3 removes true “dynamic filtering” (code-based request inspection/modification) and forces adblock innovation to depend on new Chrome APIs.
  • Concern that Chromium’s direction and Chrome Web Store policies will constrain extension-based blockers, regardless of Brave’s intentions.

Privacy, Logging, and Subscriptions

  • Leo’s “no chat logs” and unlinkable subscription tokens are praised but also questioned, as claims are ultimately trust-based.
  • Some see GDPR and Brave’s token scheme as “good enough” for typical users; others warn against over-trusting Brave, citing critical articles and general skepticism about “no logs” promises.
  • A commenter sketches how unlinkable tokens might work (random IDs, shuffled storage, limited entitlements), while noting zero-knowledge approaches as a more complex option.

Model Quality & UX

  • Several say that after using GPT‑4, Mixtral feels clearly weaker, though still strong compared to many open models. Others report Mixtral outperforming LLaMA 70B in their usage.
  • Use cases include local coding assistants (e.g., Dolphin Mixtral), offline document/code analysis, and “quick syntax/parameter” questions where latency matters.
  • Leo can be invoked from the browser omnibar; PDF summarization in Leo itself is unclear/not present, with Kagi’s extension suggested as an alternative.