Mistral AI Launches New 8x22B MOE Model

Original Article ↗ Hacker News Discussion ↗

Mistral AI has released Mixtral-8x22B, a massive open-weight Mixture-of-Experts language model (~262 GB) that targets GPT‑4-level capabilities while remaining self-hostable. Commenters focus on practical issues like hardware requirements, quantization options, and runtimes (from MacBooks to multi‑GPU rigs), as well as how its performance compares with GPT‑4, LLaMA and other open models. The release also revives questions about open vs. closed model strategy, licensing for commercial use, and potential security risks in downloading large model weight files.

Model release details

  • New Mixtral 8x22B is a Mixture-of-Experts (MoE) model, distributed via torrent; size ~262 GB (fp16), max context ~65k tokens.
  • Some confusion whether this is “Mistral Large”; commenters note sequence length differs from the known Mistral Large model.
  • Parameter counting for MoE is debated; rough back-of-the-envelope estimates suggest ~140B parameters, but exact structure is unclear.
  • Later confirmation in the thread that the model is Apache 2.0 licensed.
  • The released checkpoint is a base model, not instruct-tuned, though several report it already chats “surprisingly well.”

Hardware requirements & running tools

  • Unquantized model is too large for typical consumer GPUs; discussion centers on 3–4 bit quantization.
  • 4-bit estimates: ~70–90 GB for weights; 3-bit ~50 GB. People report running quantized versions on 96–192 GB Apple Silicon machines and large-CPU systems (Epyc/Threadripper).
  • MoE reduces compute per token but not weight memory: all experts must still be loaded, so you don’t save VRAM, only FLOPs.
  • Suggested runtimes: vLLM, llama.cpp (can also quantize), llamafile, LM Studio, plus cloud providers and together.ai playground. Splitting across multiple GPUs or CPU+GPU is common.

Performance and comparisons

  • Strong expectations based on Mixtral 8x7B, which many consider the best model for a single ~48 GB GPU.
  • Some claim open models are now at least at first-release GPT-4 level; others strongly disagree, saying even very large open models still lag GPT-4 and are closer to GPT-3 on general tasks and non-English languages.
  • A linked unofficial benchmark shows Mixtral 8x22B competitive but GPT-4 still well ahead where reported.
  • Command-R+ is noted as beating an older GPT-4 variant on one public benchmark, but is non-commercially licensed; several expect Mixtral 8x22B to be in that tier or better.

Licensing, strategy, and openness

  • Debate over “non-commercial” (e.g., CC-BY-NC) terms: private home use is clearly allowed; government, contractors, and subscription apps are legally ambiguous, and commenters say courts differ by jurisdiction.
  • Speculation on why some Mistral models are open and others closed: funding, training-data licenses, fine-tune business model, or basing some closed models on others’ checkpoints.

Security and release process

  • Some worry about a Twitter-only magnet drop with no blog/model card and potential account compromise or malicious weights.
  • Others note this “magnet first, docs later” pattern is standard for these releases.
  • Discussion of model-file supply-chain risk: past exploits via unsafe formats (e.g., pickles, older formats, GGUF bug); safetensors seen as safer but not guaranteed.

Use, alignment, and Auto-GPT

  • Advocates emphasize advantages over GPT APIs: local control, no remote data collection, and less aggressive alignment/filtering.
  • Skepticism from some who view the model as clearly inferior to upcoming LLaMA and mostly marketing; others counter that earlier Mixtral already beat LLaMA 2 70B for them.
  • One commenter describes a working multi-agent/auto-GPT-like system built with GPT-4 and plans to add Mistral support; others are unsure such systems are useful beyond toy tasks.