Mistral NeMo

Original Article ↗ Hacker News Discussion ↗

Mistral NeMo, a new 12B-parameter open-source language model co-developed by Mistral and NVIDIA, is drawing attention for its large 128k-token context window, FP8-aware training, and Apache 2.0 licensing that makes it attractive for both research and commercial use. Commenters weigh its practical hardware requirements—how well it fits on consumer GPUs, prospects for local deployment via tools like Ollama or llama.cpp, and the trade-offs of context size versus VRAM—against competing models such as Llama 3 and Gemma 2, noting some “parameter creep” in benchmark comparisons. The model’s new Tekken tokenizer, fine-tuning potential, and implications for business models around open releases also feature prominently, alongside questions about alignment policies and the real-world value of current evaluation benchmarks.

Model overview & positioning

  • Mistral NeMo is a 12B-parameter model co-developed with Nvidia, Apache 2.0–licensed, with a 128k-token context window and FP8-aware training for efficient inference.
  • It’s marketed as state-of-the-art in its size class, outperforming Llama 3 8B and similar/open models on vendor benchmarks, though some commenters note it has ~50% more parameters than 8B competitors (“parameter creep”).
  • Some see 12B as a “sweet spot” for local use: more capable than 7–8B, still feasible on consumer hardware.

VRAM, quantization, and local use

  • Rough rules of thumb discussed:
    • ~1 GB VRAM per billion params at 8-bit; ~2 GB/B at 16-bit, plus 20–40% overhead.
    • FP8-aware training aims for good quality at ~1 byte/parameter; still need headroom for KV cache, especially with 128k context.
  • Reports:
    • 8 GB VRAM is insufficient; 3060 Ti users hit OOM.
    • 4090 (24 GB) can load the full model but may OOM on multi-turn chats in some setups.
  • Quantized (4–8 bit) variants should run on 12–16 GB GPUs; some users target MacBooks with large unified memory or free Colab T4s with 4-bit QLoRA fine-tuning.
  • Expect support via llama.cpp/OLLAMA/LM Studio, but tokenizer differences (Tekken vs SentencePiece) mean extra work; not fully plug-and-play yet.

Tokenizer & multilingual behavior

  • NeMo introduces “Tekken,” a tiktoken/BPE-based tokenizer over 100+ languages, claimed to compress better than previous SentencePiece setups.
  • Discussion clarifies SentencePiece is a library that can also use BPE; the switch likely concerns engineering traits (latency, implementation) rather than fundamental compression gains.
  • Several comments discuss multilingual “cross-over”: evidence and anecdotes suggest models can transfer facts across languages, but with quirks like the “reversal curse.”

Use cases, alignment, and business questions

  • Coding remains the most emphasized specialized use case; other domains like legal/finance are seen as slower to adopt and more liability-sensitive.
  • Some criticize alignment/safety layers on “open” models; others argue they’re necessary for reputational and legal reasons.
  • Debate over open release: expensive training on thousands of H100s vs giving away weights under Apache 2.0. Many argue most users still pay for hosted convenience; others fear hyperscalers will repackage and outcompete.
  • Benchmarks: some early user tests (e.g., NYT Connections game) show NeMo trailing Gemma 2 and GPT‑4o mini; several commenters want LMSYS/arena results before judging.