Mixtral 8x7B: A sparse Mixture of Experts language model

Original Article ↗ Hacker News Discussion ↗

An open-weight language model called Mixtral 8x7B is drawing attention for using a sparse Mixture of Experts architecture to match or exceed GPT‑3.5 and Llama 2 70B on many tasks while only activating a fraction of its 47 billion parameters at inference time. Commenters focus on its practical impact: it can run locally on high-end consumer GPUs and Apple Silicon Macs with quantization, enabling hobbyist experimentation, potential in-game AI characters, and offline assistants, though VRAM and hardware costs remain constraints. The release also prompts debate over the opacity of training details in modern LLM papers, how well such models handle mathematics, the real value of informal benchmarks, and when more powerful or multimodal open models might appear.

Release and paper timing

  • Model weights (Mixtral 8x7B) were torrented in Dec 2023; the formal paper appeared later.
  • This thread focuses on the paper’s details and how they match community experience with the already‑released weights.

Architecture and capabilities

  • Mixtral is a sparse Mixture of Experts: each layer has 8 “experts,” and a router sends each token to 2 of them.
  • Effective inference use is ~13B of 47B parameters; all experts must still be loaded in memory.
  • It is reported to outperform Llama 2 70B and GPT‑3.5 on math, coding, and multilingual benchmarks; some fine‑tuned variants reportedly rival Gemini Pro.
  • Several commenters say it’s the first local model they seriously use instead of GPT‑3.5.

Hardware, quantization, and local use

  • Can run on RTX 3090 / 4090 and recent Apple Silicon (M1/M2/M3) with sufficient RAM, using 3–5 bit quantization.
  • Reports: ~50 t/s on a 3090 at 3‑bit; ~30 t/s on an M1 Max at Q4; CPU‑only is possible but slow.
  • Debate over how “consumer‑grade” this hardware is: some see 3090/large‑RAM Macs as niche and expensive; others argue used high‑VRAM GPUs are attainable for enthusiasts.
  • 32k context is advertised, but at least one user reports difficulty achieving large contexts on limited VRAM.

Applications: games and agents

  • Some see near‑term potential for in‑game NPC dialogue or strategy games with LLM‑driven leaders, either locally or via subscriptions.
  • Others argue most gamers lack the hardware; see this as “years away” for mass adoption.
  • Techniques like structured output (“grammars”) are highlighted for building agents and game‑like systems.

Math and tool use

  • Discussion on whether next‑token models can ever be truly reliable at arithmetic.
  • One side: math is inherently hard for LLMs; tools/program execution will always be needed.
  • Other side: with more scale and better training, arithmetic could emerge from better world modeling and compression.
  • Tool‑using frameworks (calculators, code execution) are mentioned as practical workarounds.

MoE expert behavior and interpretability

  • A figure in the paper shows no obvious topic‑based routing patterns; “experts” are emergent, not human‑defined.
  • Some question whether gains come from the MoE structure itself rather than clearly specialized experts.
  • The exact training procedure and datasets for experts and router are seen as under‑documented.

Multimodal and future models

  • Interest in open multimodal models; existing ones (LLaVA, CogVLM, Meta’s multimodal work) are cited, but many are non‑commercial or limited.
  • Some expect strong open models comparable to ChatGPT/GPT‑4, though others note not all future Mistral models may be fully open‑weight.

Benchmarks and paper quality

  • Informal community benchmarks are used to rank Mixtral highly, but some criticize them as non‑rigorous and non‑reproducible.
  • Broader concern that LLM evaluation is fragmented and easy to game.
  • Several commenters find the paper itself thin on training details and datasets, seeing this as part of a broader trend where companies hide key training specifics.

How to run it

  • Common tools mentioned: ollama, llama.cpp, llamafile, LM Studio, GPT4All (partial support), MLX examples.
  • Multiple step‑by‑step snippets show that a single binary (llamafile) or simple ollama pull mixtral commands are enough to try the model locally, given enough RAM/VRAM.