A messy experiment that changed how I think about AI code analysis

Original Article ↗ Hacker News Discussion ↗

Engineers react to a blog post claiming an AI system was “taught” to review code like a senior developer by restructuring how code context is fed into an LLM. Many see value in front‑loading architectural understanding, richer context, and domain-specific prompting, likening it to chain‑of‑thought for code, and note similar approaches in tools like Cursor, Aider, and custom agents. Others are skeptical, pointing out missing implementation details, cherry‑picked examples, anthropomorphic framing, and the need for rigorous evals across real-world codebases to judge whether such systems are reliably useful.

Perceived Contribution of the Technique

  • Many find the core idea useful: pre-structure the codebase, add higher-level context, and then have the LLM reason about code like a more experienced reviewer.
  • Several note this mirrors how good human reviewers triage: understand architecture and impact first, then inspect details.
  • Some see it as an example of “domain-specific chain-of-thought” prompting applied to code analysis.

Prompting, Planning, and Agentic Workflows

  • Multiple commenters already ask models to “plan first, code later” and explicitly forbid code generation until an architecture or approach is agreed.
  • Existing tools (AI IDEs, coding agents, code search systems) already implement variants of:
    • Architecture discussion / project mapping
    • Context gathering via code search and call graphs
    • Multi-file editing, build/test/deploy loops

Context, Structure, and Transformer Behavior

  • Strong agreement that more and better-curated context dramatically improves LLM outputs.
  • Some push back on “context first” language, arguing transformers see the whole window at once; others respond that ordering and scaffolding still influence behavior via prompting.

Skepticism: Missing Details, Evaluation, and Hype

  • Key functions in the article (file grouping, context extraction) are omitted, leading to accusations of “jazz hands” and hidden “secret sauce.”
  • Repeated calls for:
    • Actual source code, not just narratives
    • Benchmarks on diverse, realistic codebases and PRs
    • Metrics for correctness and significance, not just impressive anecdotes
  • Several see the tone as marketing-adjacent, typical of AI-hype content.

Junior vs Senior Analogy and Anthropomorphism

  • Heated debate over the claim that juniors read code linearly; some say this matches their early experience, others call it unrealistic and condescending.
  • The text was edited mid-thread, prompting questions about narrative reliability.
  • Many dislike anthropomorphizing AI as a “senior developer,” seeing it as misleading framing.

Real-World Use of Coding Assistants

  • Some report substantial productivity gains using tools like AI IDEs and agents for:
    • Boilerplate, stories, tests, localization, and documentation
    • Large-scale but low-conceptual work across many files
  • Others emphasize that such output still needs human review and often contains duplication or suboptimal patterns.

Limits: Hallucinations, APIs, and Verification

  • Concern that the showcased example may involve hallucinated details (e.g., fabricated PR references).
  • Common frustrations:
    • Invented APIs and mixed framework versions
    • Plausible-sounding but wrong suggestions
  • Suggestions include feeding concrete API docs, using RAG, and explicitly validating how often outputs are both correct and important.

Broader Reflections

  • Disagreement over whether tech debt is mainly a coding vs management problem.
  • Meta-discussion notes strong emotional reactions: some developers feel threatened or defensive; others accuse critics of Luddism.
  • Several see this work as one early step toward “engineering practical thinking patterns” for LLM-based tools.