Mixtral 8x22B

Original Article ↗ Hacker News Discussion ↗

Rapid progress & capabilities

  • Commenters note the pace of improvement: tasks that recently needed overnight GPU jobs now run in minutes on consumer hardware.
  • There is excitement that powerful models like Mixtral can soon run locally (after quantization), but also concern about side effects (e.g., easier deepfakes).

Costs, openness, and licensing

  • Training from scratch is debated: some claim “only” low six figures for smaller MoE models; others estimate $1M–$10M+ for production‑grade models like 8x22B, with high risk of failed runs.
  • Many argue these models are “open weights” but not “truly open,” since training data and full reproducibility are missing.
  • Liability around training data is highlighted; some expect companies to delete datasets to limit legal exposure.
  • The permissive license is widely praised and contrasted with non‑commercial licenses, which are not seen as “permissive.”

Performance, benchmarks, and GPT‑4 comparisons

  • Mixtral 8x22B is regarded as state‑of‑the‑art among permissive models and competitive with Command R+; some leaderboard results put these near GPT‑4 on certain tasks.
  • Others are skeptical: vendor benchmarks are compared to advertising, and subjective experience still favors GPT‑4.
  • Multiple independent benchmarks (Chatbot Arena, MMLU, AlpacaEval, custom puzzles) are recommended; no single benchmark is seen as definitive.

Mixture‑of‑Experts (MoE) discussion

  • Several explanations describe MoE as adding multiple MLP “experts” per layer plus a learned router that activates only a subset per token.
  • “Experts” are not human‑understandable specializations; routing is learned, often to balance load.
  • Main benefit: lower compute per token for a model with very large total parameters; memory usage still requires storing all experts.

Hardware, RAM/VRAM, and local deployment

  • 4‑bit quantized Mixtral 8x22B is reported at ~80 GB, generally beyond “most laptops,” though some claim it can run on large‑RAM machines or fast CPUs with lots of system RAM.
  • Rough rule of thumb mentioned: ~1 GB RAM per billion parameters for local CPU inference.
  • There’s significant lament about non‑upgradeable laptop RAM (especially Macs), alongside discussion of LPDDR vs SO‑DIMM and emerging modular standards.

Tools, APIs, and JSON/function calling

  • For self‑hosting, Ollama, llama.cpp servers, vLLM, and similar tools are discussed; many expose OpenAI‑compatible APIs.
  • For remote access, people mention Mistral’s own API, OpenRouter, Perplexity, and other playgrounds.
  • JSON‑constrained output is noted as a platform feature; separate small models fine‑tuned for JSON/function‑calling are recommended for self‑hosting.

Use cases for local / open models

  • Personal productivity without internet distractions (coding help, Q&A, pointers to docs).
  • Local search over personal data (emails, Common Crawl subsets, podcasts via Whisper plus LLM).
  • Role‑playing/storytelling apps, custom “chat with a deceased friend” using past conversations, and voice‑controlled desktop assistants.
  • Many value privacy, offline capability, and the ability to run uncensored models.

Context windows & long‑context research

  • Some wish Mixtral matched 128k‑token input like GPT‑4 Turbo; others argue once windows are “big enough,” exact size matters less.
  • Concerns: models often neglect the middle of long contexts and APIs still cap output length.
  • Participants reference recent work on linear/long‑range attention and speculate about future “effectively unlimited” context.

Base vs instruct models & tag confusion

  • Several clarify that base models are trained purely for next‑token prediction and are poor at following instructions without further fine‑tuning.
  • Mixtral 8x22B has both base and instruct versions; confusion arose when a local tool’s mixtral:8x22b tag initially pointed to the base model, producing odd, ad‑like continuations.
  • The tag was later switched to the instruct model, prompting concern about silent breaking changes and reminders to pin exact versions for reproducibility.

Business and ecosystem perspectives

  • Startup founders describe a “tailwind”: each new model improves cost, reliability, and context windows, making products better without extra work.
  • Others warn that thin wrappers around public LLMs are fragile businesses; real value must come from domain expertise, workflows, data, and UX, echoing the trajectory seen in image recognition.
  • There is debate over whether AI products are mainly model‑driven or primarily about surrounding features and services.

Openness, Microsoft partnership, and philosophy

  • Some view this release as evidence Mistral is still strongly committed to open weights, despite partnering with a major cloud vendor.
  • Others note subtle wording changes on the company site (from “committing to open models” to “open‑weights models”), interpreting this as a softened stance.
  • A few draw analogies to past “open source vs free software” debates, arguing current “open” AI branding can obscure important freedoms around data, reproducibility, and downstream use.