How we made our AI code review bot stop leaving nitpicky comments

Original Article ↗ Hacker News Discussion ↗

An AI-powered code review bot that overwhelmed developers with nitpicky comments prompted experimentation with prompts, LLM self-scoring, and ultimately an embedding-based KNN filter trained on which comments humans actually addressed. Commenters debate whether this kind of heuristic, trial-and-error ML approach is robust, suggesting alternatives like classical classifiers, better prompting, or simply relying on linters and human judgment. The thread reflects broader skepticism about AI in code review—its tendency to hallucinate, enforce subjective style, and distract juniors—alongside a more optimistic view of such tools as a first-pass “extra pair of eyes” rather than a replacement for human review.

Approach to reducing nitpicks (embeddings & KNN)

  • Many commenters find the final solution (embedding comments and doing KNN-style similarity filtering) plausible, even if “hacky.”
  • Some note this is effectively a simple classifier; suggest trying other ML models (random forest, XGBoost, small neural nets) on top of embeddings.
  • Idea of a “universal nit” via averaging embeddings across customers is proposed; authors say they’ll try it and already combine upvoted/downvoted sets to reduce false positives.
  • Concern raised that clustering might incorrectly suppress comments about specific modules/classes if many prior comments there were downvoted.

Prompting vs post-hoc filtering

  • Several argue the problem “should” be solvable via better prompting, including:
    • Clearer definitions instead of “nits” (e.g., “stylistic/pedantic/trivial comments”).
    • Explicit severity labels at the end of responses.
    • Chain-of-thought plus tagging nitpicks for removal in a second pass.
  • Others report similar experiments with severity scores and LLM-as-judge that still misclassified important issues as nitpicks.
  • Discussion of known failure modes: action bias, long-context confusion, ambiguous wording, conflicting instructions.

What counts as a nitpick?

  • Strong disagreement on whether the article’s example is actually a nitpick; some see it as important for long-term maintainability.
  • Many emphasize that nitpickiness is context- and company-dependent, and even the same comment can be trivial in one PR and crucial in another.
  • Some suspect ego or “ship fast” culture may drive pressure to label valid criticism as nitpicking.

Usefulness of AI code review bots

  • Mixed views:
    • Supporters see value as a first-pass “extra pair of eyes” that catches style, duplication, and obvious problems before human review.
    • Critics report high noise, hallucinated issues, and little real benefit compared to linters and human review; fear juniors over-trusting AI feedback.
    • Several argue code review is precisely where human judgment, knowledge-sharing, and mentoring are most important.

Linters, alternatives, and metrics

  • Debate whether AI review adds more than well-tuned linters/formatters; proponents point to more nuanced, context-aware rules.
  • Others say overly complex rules are themselves the problem.
  • Metric choice criticized: “percentage of comments addressed” may reward leaving fewer comments; suggestions include normalizing by files or lines changed.

Pricing & incentives

  • Some see the quoted per-file/per-dev pricing as expensive, especially for lower-wage markets or cash-strapped orgs; others note it’s a small fraction of developer salary.
  • Commenters highlight that LLMs being billed per token may bias toward verbosity, though competition and user instructions can push toward concision.