Mixtral of experts

Original Article ↗ Hacker News Discussion ↗

Mistral’s new Mixtral 8x7B model, a sparse “mixture of experts” architecture, is drawing attention for matching or surpassing GPT‑3.5 on many benchmarks while running at roughly the cost and speed of a 12–13B parameter model. Commenters explore how MoE works technically, its trade‑offs in VRAM use, inference speed, and fine‑tuning, and why open‑weight models that can be self‑hosted are strategically important for enterprises and especially European actors wary of dependence on US cloud providers. There is broad interest in Mixtral’s permissive licensing and light moderation, but also skepticism about marketing claims versus real‑world capability, context window limits, and whether such efforts can realistically challenge proprietary giants like GPT‑4 and Gemini.

Why comparisons are to GPT‑3.5, not GPT‑4

  • GPT‑3.5 is widely used, free in ChatGPT, and cheaper/faster via API; many apps target its quality/cost point.
  • Multiple open‑weight models already match or beat base 3.5 on some tasks, especially when fine‑tuned.
  • Rumored GPT‑4 scale (trillion‑param MoE) and closed access make it a different class; also nobody can self‑host it.

MoE architecture and trade‑offs

  • Mixtral 8x7B is a sparse Mixture‑of‑Experts: 8 experts plus a small router; per token, per layer, only 2 experts are used.
  • Only the feed‑forward blocks are replicated; attention is shared, so total params (~46.7B) < 8×7B.
  • Upside: quality closer to a 40–70B dense model with compute per token like ~12–13B.
  • Downside: all experts must still reside in VRAM, so it doesn’t help if GPU memory is the bottleneck; helps mainly with throughput/latency and distributed serving.
  • Good fit for large‑scale cloud inference; less ideal for single‑GPU, VRAM‑constrained setups.

Performance, benchmarks, and context

  • Mixtral (and the closed “mistral‑medium”) are reported to beat GPT‑3.5 and Llama‑2 70B on many public benchmarks, but posters stress benchmark limitations and small score deltas.
  • Some expect similar gains from a straightforward ~13B dense model; others see MoE as a key ingredient behind GPT‑4‑class systems.
  • Context window uses a sliding‑window approach; effective recall around 8k tokens is seen as limiting for document‑heavy workflows.

Open weights, censorship, and safety

  • Weights are released under a permissive license (“open weights” rather than fully open source).
  • Base models are minimally moderated and will follow unsafe instructions unless explicitly preference‑tuned or wrapped with guardrails.
  • Several see a split market: uncensored base models for research/local use vs. customized, safety‑tuned models for corporate production.

Business model, ecosystem, and hosting

  • Many see strong demand for on‑prem / private models for IP, compliance, and data‑governance reasons.
  • Mixtral is viewed as a strong European contender; some expect government and strategic funding.
  • Concern: open releases let competitors reuse innovations; counterpoint: small open models build mindshare, while larger proprietary models and hosted APIs can be monetized.
  • Local use today is possible via GGUF/llama.cpp etc., but VRAM and RAM requirements remain high; consumer hardware can run heavy quantized variants slowly.