Exploiting the most prominent AI agent benchmarks

Overall reaction to the paper

• Many commenters find the work a valuable exposé on how current agent benchmarks can be “solved” via exploits without doing the intended tasks.
• Others see it as overhyped, arguing these are ordinary software misconfigurations or interface bugs that should be GitHub issues, not a major research result.

Nature and significance of the exploits

• Exploits range from trivial (e.g., always passing because of lax scoring) to more involved (e.g., modifying wrappers or config files to run arbitrary code, downloading answer keys, self-deleting payloads).
• Some view the more advanced exploits (e.g., privilege escalation and self-cleanup) as more impressive than what the benchmarks are trying to measure.
• Others argue it’s unsurprising that, if the agent can touch the evaluation environment, it can corrupt its own score.

Benchmark design, Goodhart’s law, and incentives

• Commenters repeatedly invoke Goodhart’s law and related ideas: once a metric becomes a target, it gets gamed.
• Historical analogies are drawn to CPU/GPU and smartphone benchmark cheating.
• Debate over whether AI companies primarily want honest capability signals vs. marketing “ad copy.” Some argue internal accuracy is necessary; others think gaming is inevitable given incentives.

Training-set contamination and specific benchmarks

• Strong skepticism toward benchmarks based on public data like SWE-bench, which almost certainly sits in training corpora.
• Some note newer variants (e.g., using fresh, private problem sets) try to mitigate contamination but may still be vulnerable.

Proposed fixes and alternative evaluation approaches

• Suggested improvements: sandboxing agents, isolating harness code and answer sets, per-task fresh sandboxes, fuzzing benchmarks, and penalizing guessing.
• Emphasis that automatic scoring isn’t enough; humans must occasionally inspect whether solutions actually solve tasks vs. exploit the harness.
• Several suggest maintaining application-specific, private benchmarks and longitudinal trackers, rather than relying on public leaderboards.

Trust, cheating, and the “honor system”

• Widespread agreement that benchmarks ultimately rest on trust in the reporting organization and methodology.
• Some stress that if a lab truly wanted to cheat, it could fabricate numbers outright; exploiting harness bugs is just one failure mode.

Reaction to the blog post itself

• Multiple commenters complain the blog appears AI-written and stylistically grating.
• Some are frustrated by undisclosed AI authorship; others argue AI-generated writing is now unavoidable.