Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

Original Article ↗ Hacker News Discussion ↗

An open-source “SWE-agent” from Princeton that claims to automatically fix about 12% of Python GitHub issues has prompted mixed reactions. Commenters are intrigued by rapid progress in agentic coding tools and ideas like prompt optimization and safer sandboxes, but many argue the low success rate, chaotic behavior, and risk of low-quality or insecure code make it impractical for real-world use. The exchange broadens into whether such tools will meaningfully replace software engineers or simply shift their work toward supervising, debugging, and refining AI-generated changes.

Evaluation and Reported 12% “Success”

  • Many see a 12% fix rate as far too low for production, arguing 88% wrong fixes would swamp reviewers.
  • Others frame it as strong research progress (0 → 12% in ~2 years) and expect rapid improvement, especially on trivial bugs.
  • Some ask for concrete examples of passed/failed tests and diffs, not just aggregate metrics.

Code Quality, Benchmarks, and Representativeness

  • Strong concern that AI-generated fixes (here and in similar tools like Devin) are often “garbage,” hard to maintain, or introduce new bugs.
  • Several note SWE-bench is Python-only and not representative of real-world diversity in languages and frameworks.
  • There are accusations that benchmarks are tuned to yield exciting numbers; some claim many “fixes” are trash on inspection.

Real-World Bug Reports vs Benchmark Issues

  • Demo bugs are viewed as unusually clean, well-specified math or logic issues; real bug reports are typically vague (“I clicked X, Y happened”).
  • Users question what this proves if the bug cause is already well-known, and note such tickets would rarely be filed instead of just fixed.
  • It’s unclear how the system performs on messy, under-specified, or ambiguous reports.

Agent Behavior, Demos, and Interfaces

  • The demo UI is criticized as slow and gimmicky; some suspect cherry-picked scenarios (e.g., Sympy tasks reminiscent of other demos).
  • The Agent-Computer Interface (scrolling, partial file views) is seen as an interesting workaround for context window limits.

Developer Workflow and Review Burden

  • Key open question: in the 88% of failures, is any work salvageable, or is it just wasted tokens and reviewer time?
  • Some suggest guardrails: sandboxed changes, diff-review UIs, versioning, and easy rollback to manage chaotic agent behavior.
  • A frequent worry: if AI-generated PRs become common, public issue trackers and maintainers could be overwhelmed by low-quality patches.

Current Coding Capabilities and Limits

  • Long Rust UTF‑8 example thread shows LLMs can produce compiling, mostly-correct code with guidance, but:
    • They hallucinate APIs, miss edge cases, and often choose inefficient or unsafe designs.
    • Human supervision is required to refine requirements, check performance, error handling, and correctness.
  • Several argue the hard parts of SWE are debugging, dealing with ambiguity, and value judgments, not typing code.

Job Impact and Future Speculation

  • Opinions diverge: some see SWE roles transforming into “AI herders” within 5–10 years; others think full replacement is unlikely in their lifetimes.
  • There’s discussion of recursive improvement (models engineering better models) versus skepticism that current systems can simulate senior engineers.

Security and Ecosystem Risks

  • Concern that the agent runs arbitrary shell commands based on untrusted input, inviting escapes or supply-chain attacks.
  • Fears that automated fixes could quietly introduce vulnerabilities; comparisons are made to recent high-profile backdoor incidents.
  • Some predict that if AI PRs become cheap and common, maintainers might shut down public trackers due to review cost and spam.

Alternative Uses and Tooling Ideas

  • Suggestions include:
    • Agents to help users write better bug reports instead of fixing them directly.
    • Tools to auto-find modernization opportunities in OSS (e.g., Python packaging) rather than blind bug fixing.
    • Multi-agent or “business analyst” layers to clarify requirements, though stacking AI logic is reported to be fragile in practice.