Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

Original Article ↗ Hacker News Discussion ↗

Evaluation and Reported 12% “Success”

  • Many see a 12% fix rate as far too low for production, arguing 88% wrong fixes would swamp reviewers.
  • Others frame it as strong research progress (0 → 12% in ~2 years) and expect rapid improvement, especially on trivial bugs.
  • Some ask for concrete examples of passed/failed tests and diffs, not just aggregate metrics.

Code Quality, Benchmarks, and Representativeness

  • Strong concern that AI-generated fixes (here and in similar tools like Devin) are often “garbage,” hard to maintain, or introduce new bugs.
  • Several note SWE-bench is Python-only and not representative of real-world diversity in languages and frameworks.
  • There are accusations that benchmarks are tuned to yield exciting numbers; some claim many “fixes” are trash on inspection.

Real-World Bug Reports vs Benchmark Issues

  • Demo bugs are viewed as unusually clean, well-specified math or logic issues; real bug reports are typically vague (“I clicked X, Y happened”).
  • Users question what this proves if the bug cause is already well-known, and note such tickets would rarely be filed instead of just fixed.
  • It’s unclear how the system performs on messy, under-specified, or ambiguous reports.

Agent Behavior, Demos, and Interfaces

  • The demo UI is criticized as slow and gimmicky; some suspect cherry-picked scenarios (e.g., Sympy tasks reminiscent of other demos).
  • The Agent-Computer Interface (scrolling, partial file views) is seen as an interesting workaround for context window limits.

Developer Workflow and Review Burden

  • Key open question: in the 88% of failures, is any work salvageable, or is it just wasted tokens and reviewer time?
  • Some suggest guardrails: sandboxed changes, diff-review UIs, versioning, and easy rollback to manage chaotic agent behavior.
  • A frequent worry: if AI-generated PRs become common, public issue trackers and maintainers could be overwhelmed by low-quality patches.

Current Coding Capabilities and Limits

  • Long Rust UTF‑8 example thread shows LLMs can produce compiling, mostly-correct code with guidance, but:
    • They hallucinate APIs, miss edge cases, and often choose inefficient or unsafe designs.
    • Human supervision is required to refine requirements, check performance, error handling, and correctness.
  • Several argue the hard parts of SWE are debugging, dealing with ambiguity, and value judgments, not typing code.

Job Impact and Future Speculation

  • Opinions diverge: some see SWE roles transforming into “AI herders” within 5–10 years; others think full replacement is unlikely in their lifetimes.
  • There’s discussion of recursive improvement (models engineering better models) versus skepticism that current systems can simulate senior engineers.

Security and Ecosystem Risks

  • Concern that the agent runs arbitrary shell commands based on untrusted input, inviting escapes or supply-chain attacks.
  • Fears that automated fixes could quietly introduce vulnerabilities; comparisons are made to recent high-profile backdoor incidents.
  • Some predict that if AI PRs become cheap and common, maintainers might shut down public trackers due to review cost and spam.

Alternative Uses and Tooling Ideas

  • Suggestions include:
    • Agents to help users write better bug reports instead of fixing them directly.
    • Tools to auto-find modernization opportunities in OSS (e.g., Python packaging) rather than blind bug fixing.
    • Multi-agent or “business analyst” layers to clarify requirements, though stacking AI logic is reported to be fragile in practice.