Benchmarking Opus 5 on SlopCodeBench
Benchmark results from SlopCodeBench suggest that Anthropic’s new Claude Opus 5 model offers only modest gains over Opus 4.x on multi-step coding tasks, while often generating more complex, “sloppier” code. Commenters argue that current LLMs still struggle with long-term maintainability, refactoring, and avoiding unnecessary abstractions, and that harness design and prompts can be as important as the underlying model. Many call for better benchmarks, human baselines, and RL signals that explicitly reward simplicity and code health, especially as some developers report preferring older models or Anthropic’s Fable for complex, sustained work.
Overall reaction to Opus 5 vs earlier models
- Many say Opus 5 feels only mildly better than Opus 4.8, not a “wow” jump.
- Some prefer 4.8’s behavior and tone; Opus 5 is described as overconfident, verbose, and scholastic, generating “slop” and pedantic reviews.
- A minority report real productivity gains, especially using Opus 5 medium as a faster, cheaper replacement for 4.8 x-high, with Fable reserved for the hardest tasks.
SlopCodeBench and code maintainability
- SlopCodeBench is praised as a rare benchmark that tests longitudinal behavior: multiple feature additions and how code quality evolves.
- Opus 5’s ~24% strict pass rate vs ~17% for Opus 4.6 is noted; some view that as a modest but real gain, others as still unacceptably low.
- Key concern: models add lots of functions and complexity over time; current RL/benchmarks rarely penalize complexity, so models don’t learn to simplify.
Harnesses, prompts, and agent workflows
- Several argue “slop” is often a harness/system‑prompt issue: agents with too much freedom and no constraints on where to edit accumulate mess.
- Static “skills” or templated workflows sometimes hurt performance on SlopCodeBench, likely by wasting context and forcing suboptimal workflows on greenfield tasks.
- Others report success with:
- Constraining edits to narrow seams.
- Periodic dedicated “refactor turns” or whole‑codebase review passes.
- Explicit instructions preferring simplicity and DRY code in project configuration files.
Model behavior, degradation, and benchmarking
- Some users feel models (including Fable) degrade after launch, possibly from cost‑driven changes like quantization; others cite public trackers showing mostly monotonic improvement.
- There is concern that labs could special‑case benchmark-like inputs, making public benchmarks less trustworthy.
- Participants call for more robust eval suites (including maintainability metrics) and note that most teams lack the expertise to build them.
Does “slop” actually matter?
- One thread asks whether ugliness/complexity matters if defects stay low and clients are happy.
- Replies link this directly to long‑term change cost and AI agent efficiency: messy code is harder for both humans and models to modify.
- Recent papers (mentioned but not detailed) reportedly find “cleaner” code improves agent navigation and efficiency.