SWE-bench Verified no longer measures frontier coding capabilities

Original Article ↗ Hacker News Discussion ↗

AI coding benchmarks like SWE-bench Verified are coming under fire for being contaminated by training data and riddled with flawed test cases, making high scores increasingly reflect memorization and overfitting rather than real software engineering skill. Commenters argue that once a benchmark is public and financially important, labs inevitably game it, and that underspecified or brittle tests can unfairly penalize valid solutions while rewarding models that have simply seen the answers before. Many call for new approaches such as private or rotating benchmarks, more realistic multi-step and agentic tasks, and domain-specific internal evals that better match real-world coding workflows.

Localization / UX Tangent

  • Several comments complain about forced automatic translation of the OpenAI page and other apps, with no obvious way to switch back to English.
  • People argue users should always be able to override language, independent of headers or IP-based geolocation.

Flaws in SWE-bench Verified

  • OpenAI’s audit claims a large fraction of frequently-failed tasks have flawed or overly strict tests (e.g., requiring specific function names or implementations not specified in the task).
  • Commenters calculate this implies roughly one in six tasks is problematic, which many see as “extraordinarily” high.
  • Others note this is still acceptable if the benchmark’s main job is ranking models relative to each other, not providing an absolute quality measure.

Contamination and Goalpost Moving

  • Many highlight contamination: models can reproduce exact bug fixes or problem statements, proving benchmark items are in training data.
  • Some see OpenAI’s move away from SWE-bench Verified as “moving the goalposts”; others say that’s inevitable and healthy once a benchmark is saturated or compromised.
  • There is skepticism about all major labs avoiding benchmark leakage, given strong marketing incentives.

Interpreting High Scores (Opus, Mythos, etc.)

  • Tension arises because some models report ~90%+ scores on a benchmark OpenAI now calls unreliable.
  • Possible explanations discussed: contamination, benchmark-specific optimization (“benchmaxxing”), better “guessing” of repo-specific styles, or issues with OpenAI’s own audit.
  • Several note you can’t distinguish recall from reasoning at very high scores; a model at 93% vs 90% might just memorize more.

Broader Benchmark Critiques

  • Benchmarks are seen as narrow, easily gamed, and quickly outdated once public; comparisons to ImageNet mislabels, database “benchmarketing,” and SPEC.
  • Many advocate private or “blind” benchmarks, rotating or Olympiad-style test sets, or domain-specific hidden suites.
  • Others argue static pass/fail coding tests miss what matters: integration into real workflows, long-context robustness, and agentic behavior.

Alternatives and New Directions

  • Mentioned alternatives include SWE-bench Pro and follow-ons, ARC-AGI, game-based tests (Zork, StarCraft/Go), agent benchmarks, and third-party suites (e.g., coding/problem-solving evals).
  • Some report stagnation in real-world coding quality despite large benchmark gains, suggesting overfitting to tests rather than genuine capability growth.