Mistral AI Launches New 8x22B MOE Model

Original Article ↗ Hacker News Discussion ↗

Model release details

  • New Mixtral 8x22B is a Mixture-of-Experts (MoE) model, distributed via torrent; size ~262 GB (fp16), max context ~65k tokens.
  • Some confusion whether this is “Mistral Large”; commenters note sequence length differs from the known Mistral Large model.
  • Parameter counting for MoE is debated; rough back-of-the-envelope estimates suggest ~140B parameters, but exact structure is unclear.
  • Later confirmation in the thread that the model is Apache 2.0 licensed.
  • The released checkpoint is a base model, not instruct-tuned, though several report it already chats “surprisingly well.”

Hardware requirements & running tools

  • Unquantized model is too large for typical consumer GPUs; discussion centers on 3–4 bit quantization.
  • 4-bit estimates: ~70–90 GB for weights; 3-bit ~50 GB. People report running quantized versions on 96–192 GB Apple Silicon machines and large-CPU systems (Epyc/Threadripper).
  • MoE reduces compute per token but not weight memory: all experts must still be loaded, so you don’t save VRAM, only FLOPs.
  • Suggested runtimes: vLLM, llama.cpp (can also quantize), llamafile, LM Studio, plus cloud providers and together.ai playground. Splitting across multiple GPUs or CPU+GPU is common.

Performance and comparisons

  • Strong expectations based on Mixtral 8x7B, which many consider the best model for a single ~48 GB GPU.
  • Some claim open models are now at least at first-release GPT-4 level; others strongly disagree, saying even very large open models still lag GPT-4 and are closer to GPT-3 on general tasks and non-English languages.
  • A linked unofficial benchmark shows Mixtral 8x22B competitive but GPT-4 still well ahead where reported.
  • Command-R+ is noted as beating an older GPT-4 variant on one public benchmark, but is non-commercially licensed; several expect Mixtral 8x22B to be in that tier or better.

Licensing, strategy, and openness

  • Debate over “non-commercial” (e.g., CC-BY-NC) terms: private home use is clearly allowed; government, contractors, and subscription apps are legally ambiguous, and commenters say courts differ by jurisdiction.
  • Speculation on why some Mistral models are open and others closed: funding, training-data licenses, fine-tune business model, or basing some closed models on others’ checkpoints.

Security and release process

  • Some worry about a Twitter-only magnet drop with no blog/model card and potential account compromise or malicious weights.
  • Others note this “magnet first, docs later” pattern is standard for these releases.
  • Discussion of model-file supply-chain risk: past exploits via unsafe formats (e.g., pickles, older formats, GGUF bug); safetensors seen as safer but not guaranteed.

Use, alignment, and Auto-GPT

  • Advocates emphasize advantages over GPT APIs: local control, no remote data collection, and less aggressive alignment/filtering.
  • Skepticism from some who view the model as clearly inferior to upcoming LLaMA and mostly marketing; others counter that earlier Mixtral already beat LLaMA 2 70B for them.
  • One commenter describes a working multi-agent/auto-GPT-like system built with GPT-4 and plans to add Mistral support; others are unsure such systems are useful beyond toy tasks.