Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
Benchmarking AI coding agents on licensed, private enterprise codebases is stirring interest and skepticism: one new “Real-SWE” benchmark finds even top models only solve about a third of long-horizon software tasks, with Fable, Astra, and Gemini 3.8 Flash leading. Commenters debate how meaningful such opaque, non-reproducible tests are, how much results depend on the harness and workflow rather than model quality alone, and whether sending proprietary code to vendors undermines privacy claims. Many report that while agents can be powerful on well-scoped, boilerplate-heavy work, they still struggle with business logic, prioritization, and design quality in complex real-world systems.
Benchmark design & purpose
- Benchmarks real-world software engineering tasks on large, private production codebases licensed from companies (some with many users, including popular apps).
- Tasks are “long-horizon” engineering efforts that real engineers spent weeks on, not LeetCode-style problems.
- Models are evaluated on success rate, with additional metrics like “missed requirements,” “unverified assumptions,” and “integration errors.”
- Authors plan to open-source some tasks and trajectories; all models run with “high reasoning” and mostly provider-native harnesses.
Reproducibility, transparency, and trust
- Several commenters criticize the non-public codebases: results are not easily reproducible; you must “trust the tester.”
- Others argue private benchmarks are necessary to prevent overfitting/“benchmaxxing” and are common already.
- Some worry about whose codebases are used and whether serious companies would really allow them to be fed to many LLMs; others note abandoned or licensed products may make this plausible.
- Concern that once code and prompts are sent to providers, they are no longer truly private.
Model rankings vs real-world experience
- Reported ranking: Fable 5.1 ≈ Astra > Gemini 3.8 Flash > GLM 5.3 > Grok 4.6/Muse Spark > Kimi K3 > GPT‑5.6 Sol.
- Many say Gemini 3.8 Flash’s strong score matches their experience (fast, good on large codebases); others report it being slow, tool-call-happy, or “terrible.”
- Some feel Sol and Astra are much better than the benchmark suggests; others find Sol underwhelming or fragile (e.g., mixing “thinking” into output on big projects).
- GLM 5.3 scoring above Sol surprises some who see it as more mid-tier.
- Several note these results diverge from other benchmark sites and personal use; methodology details are seen as incomplete.
Harnesses, agents, and workflow effects
- Many emphasize the “wizard, not the wand”: user skill, harness design, code maps/“atlas,” and test harnesses greatly affect outcomes.
- Provider CLIs and third-party harnesses (e.g., Antigravity, oh-my-pi) significantly change behavior, token usage, and speed.
- Some combine multiple models in orchestrated roles (planner vs implementer vs verifier) and see big differences vs single-model use.
Limits of current agents
- Common failures: missed requirements, unverified assumptions, and business-logic gaps, especially outside CRUD-like tasks.
- Overengineering, random unrequested changes, and poor design are frequent complaints even when code “works.”
- Several users have built similar private benchmarks on their own codebases, both to compare models and to reduce dependency anxiety on major providers.