Will It Mythos?
Claims that Anthropic’s Mythos/Fable model can autonomously uncover dangerous software vulnerabilities prompt a closer look at how current large language models actually perform at security auditing. Commenters examine a benchmark where multiple models scan real-world codebases without being told where the bugs are, finding that several open and Chinese models (like DeepSeek and MiMo) approach or match top Western systems at far lower cost, while still lagging behind Mythos’ apparent capabilities. The thread also raises concerns about safety guardrails, model “nerfing,” and how quickly autonomous security tooling could shift the balance between attackers and defenders.
Benchmark design and scope
- Corpus consists of real bugs Mythos previously found; other models are tested on those files.
- During benchmarking, models are asked to audit a file (with optional repo context) without being told where or what the bug is.
- A stronger “judge” model, given the bug location and description, scores whether contestants found and explained the right issue.
- Cost caps (e.g., $100 per model) significantly affect results; GPT‑5.5‑Pro only completed 4/9 cases before hitting the limit.
- A minimal harness with basic tools (read/grep) performed as well as or better than richer “agent” setups, while consuming fewer tokens.
Relative model performance
- No public model matched Mythos’ implied 9/9 performance; top non‑Mythos models typically found 4/9 bugs.
- GPT‑5.5‑Pro surfaced at the top by percentage only because it ran fewer cases and is considered unrealistically expensive for broad audits.
- Follow‑on analysis using Wilson score and time/cost suggests DeepSeek‑V4 and MiMo v2.5 Pro as best value among tested models.
- Replication runs suggest Gemma 4 31B (dense) is exceptionally strong for its size, sometimes finding 6/9 bugs and rivaling larger models.
- Cheap Chinese/open models (DeepSeek, MiMo, Qwen) are seen as genuinely competitive, not just “benchmaxxed”.
Mythos/Fable vs other models
- Many report Fable/Mythos as a noticeable step above Opus and others, especially in:
- Security auditing, reverse‑engineering, and finding subtle bugs.
- Spatial reasoning and complex math/geometry (e.g., 6DOF, computational geometry).
- Autonomously driving large code changes or meta‑applications.
- Some users see smaller or task‑specific models (e.g., Codex, GPT‑5.5) outperform Fable on narrow, highly optimized workloads.
- Others felt Fable/Mythos were overhyped or only marginally better, especially when accounting for high token use.
Guardrails, safety, and “nerfing”
- Debate over whether Mythos is just a standard model with safety filters off vs a distinct fine‑tune plus specialized harness and persistence.
- Observations that some Google offerings (Gemini via Antigravity) now resist security tasks, while Gemma 4 remains strong at bug‑finding.
- Widespread perception that older frontier models (e.g., Opus 4.6, o4‑mini) have degraded over time; proposed mechanisms include quantization, reduced reasoning budgets, KV‑cache compression, and MoE expert reduction.
- Skeptics compare “nerfing” claims to audiophile placebo; others cite performance trackers suggesting systematic pre‑release drift.
Usage patterns and broader implications
- Several participants describe Fable as better at persistent, goal‑directed work but also more “agentic,” sometimes over‑autonomously editing code.
- Some prefer chatting with models in a human‑like style, claiming better results and reduced cognitive mode‑switching; others worry this blurs human–machine boundaries and risks “AI psychosis”.
- Key concern about Mythos: enabling non‑experts to find and weaponize zero‑days, though defenders can also use LLMs to discover and patch vulnerabilities.