Mixtral 8x22B

Original Article ↗ Hacker News Discussion ↗

Mistral’s release of Mixtral 8x22B, a large Mixture-of-Experts language model with open weights and permissive licensing, is being hailed as a major step forward for self-hostable AI close to GPT‑4 performance. Commenters weigh the benefits of “open” models that can be downloaded and run locally against the reality that training remains prohibitively expensive and datasets are undisclosed, limiting true reproducibility and rigorous evaluation. They also examine practical hurdles such as VRAM and RAM requirements, skepticism about vendor benchmarks, and how rapidly improving foundation models are reshaping both personal tooling and AI startup strategies.

Rapid progress & capabilities

  • Commenters note the pace of improvement: tasks that recently needed overnight GPU jobs now run in minutes on consumer hardware.
  • There is excitement that powerful models like Mixtral can soon run locally (after quantization), but also concern about side effects (e.g., easier deepfakes).

Costs, openness, and licensing

  • Training from scratch is debated: some claim “only” low six figures for smaller MoE models; others estimate $1M–$10M+ for production‑grade models like 8x22B, with high risk of failed runs.
  • Many argue these models are “open weights” but not “truly open,” since training data and full reproducibility are missing.
  • Liability around training data is highlighted; some expect companies to delete datasets to limit legal exposure.
  • The permissive license is widely praised and contrasted with non‑commercial licenses, which are not seen as “permissive.”

Performance, benchmarks, and GPT‑4 comparisons

  • Mixtral 8x22B is regarded as state‑of‑the‑art among permissive models and competitive with Command R+; some leaderboard results put these near GPT‑4 on certain tasks.
  • Others are skeptical: vendor benchmarks are compared to advertising, and subjective experience still favors GPT‑4.
  • Multiple independent benchmarks (Chatbot Arena, MMLU, AlpacaEval, custom puzzles) are recommended; no single benchmark is seen as definitive.

Mixture‑of‑Experts (MoE) discussion

  • Several explanations describe MoE as adding multiple MLP “experts” per layer plus a learned router that activates only a subset per token.
  • “Experts” are not human‑understandable specializations; routing is learned, often to balance load.
  • Main benefit: lower compute per token for a model with very large total parameters; memory usage still requires storing all experts.

Hardware, RAM/VRAM, and local deployment

  • 4‑bit quantized Mixtral 8x22B is reported at ~80 GB, generally beyond “most laptops,” though some claim it can run on large‑RAM machines or fast CPUs with lots of system RAM.
  • Rough rule of thumb mentioned: ~1 GB RAM per billion parameters for local CPU inference.
  • There’s significant lament about non‑upgradeable laptop RAM (especially Macs), alongside discussion of LPDDR vs SO‑DIMM and emerging modular standards.

Tools, APIs, and JSON/function calling

  • For self‑hosting, Ollama, llama.cpp servers, vLLM, and similar tools are discussed; many expose OpenAI‑compatible APIs.
  • For remote access, people mention Mistral’s own API, OpenRouter, Perplexity, and other playgrounds.
  • JSON‑constrained output is noted as a platform feature; separate small models fine‑tuned for JSON/function‑calling are recommended for self‑hosting.

Use cases for local / open models

  • Personal productivity without internet distractions (coding help, Q&A, pointers to docs).
  • Local search over personal data (emails, Common Crawl subsets, podcasts via Whisper plus LLM).
  • Role‑playing/storytelling apps, custom “chat with a deceased friend” using past conversations, and voice‑controlled desktop assistants.
  • Many value privacy, offline capability, and the ability to run uncensored models.

Context windows & long‑context research

  • Some wish Mixtral matched 128k‑token input like GPT‑4 Turbo; others argue once windows are “big enough,” exact size matters less.
  • Concerns: models often neglect the middle of long contexts and APIs still cap output length.
  • Participants reference recent work on linear/long‑range attention and speculate about future “effectively unlimited” context.

Base vs instruct models & tag confusion

  • Several clarify that base models are trained purely for next‑token prediction and are poor at following instructions without further fine‑tuning.
  • Mixtral 8x22B has both base and instruct versions; confusion arose when a local tool’s mixtral:8x22b tag initially pointed to the base model, producing odd, ad‑like continuations.
  • The tag was later switched to the instruct model, prompting concern about silent breaking changes and reminders to pin exact versions for reproducibility.

Business and ecosystem perspectives

  • Startup founders describe a “tailwind”: each new model improves cost, reliability, and context windows, making products better without extra work.
  • Others warn that thin wrappers around public LLMs are fragile businesses; real value must come from domain expertise, workflows, data, and UX, echoing the trajectory seen in image recognition.
  • There is debate over whether AI products are mainly model‑driven or primarily about surrounding features and services.

Openness, Microsoft partnership, and philosophy

  • Some view this release as evidence Mistral is still strongly committed to open weights, despite partnering with a major cloud vendor.
  • Others note subtle wording changes on the company site (from “committing to open models” to “open‑weights models”), interpreting this as a softened stance.
  • A few draw analogies to past “open source vs free software” debates, arguing current “open” AI branding can obscure important freedoms around data, reproducibility, and downstream use.