Senior SWE-Bench: open-source benchmark that assesses agents as senior engineers

Benchmark difficulty and human comparison

• Current top solve rate is ~24% using a frontier model; some wonder what a competent human engineer would score.
• Several note that humans can specialize in a single codebase and use LLMs as tools, so raw percentage comparisons are tricky.
• There is curiosity about how human-in-the-loop setups (LLM asking humans for help) would affect performance.

Benchmark design, objectivity, and scope

• Many criticize the benchmark as too subjective and narrow, especially around “tasteful” vs “bloated” solutions.
• Some argue benchmarks should focus on end-to-end product quality (features, bugs, behavior over time) rather than code diffs.
• Others stress that real “senior” work includes planning, documentation, requirements gathering, and organizational constraints—hard to encode in a single coding task.
• There’s a call for clearer articulation of what axes are being tested (planning, design, maintainability, performance, etc.).

LLMs as judges and the notion of “taste”

• Skepticism about having LLMs judge subjective qualities like “tasteful solves,” especially if the coding model and judging model are related.
• Concerns about bias (models preferring their own family’s outputs) and “prompt woo” like “you are a senior reviewer, make no mistakes.”
• Counterpoint: some standards (e.g., maintainability metrics, ISO-like rules) can partially objectify code quality, but even those are contested.
• Debate over whether “taste” is just ego and vibes vs a hardened System 1 heuristic for maintainability and robustness.

Model comparisons and harness effects

• Mixed experiences comparing Opus 4.8 and GPT 5.5: some see Opus as far better at underspecified, high-level work; others find GPT 5.5 clearly superior except in specific domains.
• Several emphasize that the “harness” (tools, environment, prompting) matters as much as the base model.

Underspecified requirements and what “senior” means

• The benchmark’s claim that senior engineers build features from underspecified requirements is challenged.
• Multiple commenters say real seniors proactively obtain requirements from users, metrics, and stakeholders instead of inventing them.
• Others note that seniors must also resolve ambiguity and intuit missing constraints, not just execute fully specified tasks.

Longevity, gaming, and alternative benchmarks

• Being open-source and based on real OSS changes raises concerns about overfitting and models memorizing training data.
• Questions about how to keep the benchmark comparable over time as models train on its tasks.
• Ideas floated include adversarial question generation with ELO-style scoring, human-grounded difficulty tiers, and simulated evolving requirements with LLM roleplay.