Mistral CEO confirms 'leak' of new open source AI model nearing GPT4 performance
A leaked “miqu-1-70b” large language model, traced to French startup Mistral AI, is drawing attention for reportedly approaching GPT‑4-level performance while being distributed in quantized form suitable for local use. Commenters debate whether the leak is a deliberate PR move, how much it actually matches GPT‑4 in real-world tasks, and what “open source” should mean for AI models when only weights—not training data—are released. The incident also fuels broader comparisons between Mistral and OpenAI on cost, guardrails, performance stability, and the pace of progress in proprietary versus open-weight models.
Nature of the “leak” and marketing questions
- A quantized, watermarked 70B model (“miqu-1-70b”) appeared on Hugging Face; CEO called it an older Llama-2–based model once given to early-access customers.
- Some see it as an “over‑enthusiastic employee” leak; others think it looks like deliberate, clever PR.
- Mistral hasn’t forced takedowns; instead they left a light-touch note, which many interpret as a confident, developer-friendly stance.
Model capability and comparison to GPT‑4 / GPT‑3.5
- Users running Mixtral and derivatives locally report:
- For many coding and discussion tasks, quality feels close to GPT‑4, but with more hallucinations and less self-correction.
- Another practitioner rates Mixtral-style models as ~5% GPT‑4‑level, ~75% on par with GPT‑3.5, ~20% worse (too chatty, hallucinatory).
- Mistral Medium via API is often described as comparable to GPT‑3.5; some say “between 3.5‑turbo and 4‑turbo.”
- “Near GPT‑4” is seen by some as hype; others emphasize the significance of getting close with much smaller, cheaper, often single‑GPU models.
Guardrails, “nerfing,” and preference for open models
- Multiple comments complain that OpenAI models have become less helpful due to stricter safety and liability guardrails (especially for legal/contract help and larger tasks), sometimes doing partial work or telling users to consult experts.
- Mistral’s API is praised for allowing guardrails to be disabled and feeling more “adult” and less paternalistic.
- Some are shifting usage to local/open models for privacy, stability, and to avoid policy drift.
Open source vs “open weights” and legal/IP issues
- Debate over whether this leak is truly “open source” or just “open weights.”
- Some argue open source in AI should mean free use, modification, and inspection of weights, even without training data.
- Others stress that many LLM licenses (e.g., with field-of-use restrictions) are not “open” in the traditional free‑software sense.
- Long legal thread on whether model weights are copyrightable, database‑protected, or trade secrets; outcome is acknowledged as unclear.
Watermarking and quantization details
- Watermarking ideas discussed: weight perturbations, special “shibboleth” tokens, and statistical output patterns.
- Quantization is clarified as lowering numeric precision for weights, not replacing “numeric sequences.”
- Leak came as GGUF quants (Q2/Q4/Q5); convenient for local use but seen by some as suboptimal for high‑throughput production.
Benchmarks, leaderboards, and evaluation bias
- HuggingFace/LMSys leaderboards are referenced; some note a gap between GPT‑4 “classic” and GPT‑4‑Turbo.
- Others argue leaderboards are noisy, suffer from sampling bias (dev/English‑centric queries), and risk Goodhart’s Law; real capabilities (e.g., creative writing, niche languages, long‑context uses) may be under‑measured.