Thoughts on a month with Devin
AI “software engineer” agents like Devin are impressing developers with autonomous workflows, but users report they are unreliable, hard to steer, and prone to wasteful infinite loops or subtle bugs that still demand senior-level review. Many engineers find more value in constrained tools such as Copilot, Cursor, Aider, or OpenHands that act as assistants for small, well-bounded tasks rather than replacements for humans. Commenters are split between skepticism that current agents are overhyped and not yet cost‑effective, and optimism that rapid model and tooling improvements could eventually automate a large share of routine coding work, with significant implications for how software teams are structured.
Devin and agentic workflows: strengths
- When tasks are well-scoped, stacks are mainstream, and tests are easy to run, Devin can produce clean, test‑passing code and handle multi-file changes autonomously.
- The “agent in a Slack/terminal” UX and closed-loop workflows (edit → run tests → iterate) impressed many and shifted expectations about what’s possible.
- Some see current results as analogous to early image generation: rough now, but the mere fact it works at all suggests large future upside.
Major limitations and failure modes
- Tends to make extraneous edits beyond the request, sometimes breaking unrelated functionality, and is bad at rolling those changes back.
- Often gets stuck in “infinite thinking loops,” working for hours or days instead of asking for help, especially on “soft stops.”
- Poor at admitting incapacity or integrating coaching; frequently compared to the worst stereotype of an overconfident junior dev.
- Subtle, hard‑to‑spot mistakes (e.g., silently truncating a license header) undermine trust.
- Users report no reliable way to predict which tasks it will succeed on, limiting its value as a tool.
Agents vs. narrower tools
- Many argue Devin overreaches; narrower agents focused on bug fixes, small features, test/CI cleanup, or maintenance show much higher success rates and real enterprise interest.
- Constrained agents and IDE‑integrated tools (Cursor, Copilot, Aider, OpenHands, others) are seen as more practical: they act as “power tools,” not replacements.
- There’s discussion of orchestrators, time/“energy” limits, and supervisors (even non‑LLM models) to detect when an agent is stuck and halt or escalate.
Where LLM coding helps today
- Explaining legacy or “arcane” code, proposing refactors, and writing tests with many edge cases.
- Generating small, next‑step snippets in data science, SQL, matplotlib, shell/HTTP work, and onboarding to new technologies.
- Automating tedious chores: merge conflicts, linter fixes, parameter reshuffling, multi‑file refactors.
Models, hallucinations, and trajectory
- Some report newer reasoning models (e.g., o1 variants, Claude Sonnet 3.5) hallucinate less for coding with good prompts and short context; others still feel “burned” and prefer to write code themselves.
- Debate over whether progress is still rapid or already hitting diminishing returns.
- Broad agreement that AI cannot yet replace engineers; its output needs review comparable to a brand‑new hire.
- Many expect continued pressure from companies to cut headcount using AI, with disagreement on how far that will actually go.