Measuring the sloppiness of code

Original Article ↗ Hacker News Discussion ↗

Claims that “coding is solved” by AI clash with growing concern that large language models generate vast amounts of low‑quality, hard‑to‑maintain code. Commenters explore attempts to quantify this “sloppiness” using metrics like lines of code, cyclomatic complexity, verbosity rules, and multi‑step benchmarks that show agents accumulating technical debt over iterations. Many argue that while models can often produce correct snippets quickly, true code quality—maintainability, architecture, safety, and human or agent comprehensibility at scale—remains unsolved and difficult to measure without falling prey to Goodhart’s law.

SlopCodeBench and “slop” accumulation

  • Benchmark flips usual pattern: multiple iterative tasks with context wiped between rounds, mimicking real agent use.
  • Poor early design decisions compound; under strict criteria (all checkpoints pass), even top models reportedly achieve 0% solve.
  • Simple metrics used: LOC growth, cyclomatic complexity, and heuristic “verbosity rules” to detect overlong or redundant code.
  • Many commenters appreciate finally having a quantitative handle on a phenomenon they see in practice.

What is code quality, really?

  • Several argue correctness is only a baseline; quality also includes efficiency, security, maintainability, readability, observability, portability, etc.
  • Others note that much human enterprise code has historically been low quality; AI slop is seen as “more of the same, just faster.”
  • Some claim code quality is fundamentally hard or intractable to measure (likened to the halting problem) and inherently vibe-based.

Are LLMs good coders? Conflicting experiences

  • Enthusiastic side:
    • For many tasks, LLM output is said to be better than that of an average or junior dev, especially with a competent human guiding it.
    • People report higher throughput with acceptable quality, especially for higher-level languages and greenfield work.
  • Skeptical side:
    • Agents overproduce code, duplicate logic, avoid deleting dead code, and break existing architecture.
    • They often fail at non-local changes, complex systems, and subtle bugs, or require heavy micromanagement.
    • Some say LLM-authored codebases “speedrun” themselves into unmaintainable states.

Global design, maintainability, and mental models

  • Commenters stress that the hardest problems are global: separation of concerns, layering, interfaces, and long-term evolution.
  • Coding is described as a way humans build shared mental models; if agents do all coding, humans may lose understanding and control.
  • Others argue future tooling may provide “views” over huge codebases, reducing the need for human-held global models.

Metrics, Goodhart’s law, and evaluation ideas

  • LOC change is widely viewed as a surprisingly effective smell metric but vulnerable to optimization/pathological “code golfing.”
  • Suggestions: combine LOC/tokens with cyclomatic complexity, churn, coupling, cohesion, indentation depth, AST patterns, and token cost to “grok” code.
  • Some propose:
    • Iterative “number of rounds solved correctly” as a core metric.
    • Using a separate baseline model to judge whether another model’s code is usable.
    • Domain/architecture benchmarks (e.g., progressively more complex “counter” apps) that measure design quality, not just passing tests.

Industry impact and open questions

  • Many distinguish “coding” from “software engineering”; they see the former approaching automation faster than the latter.
  • There’s debate over whether current models already surpass “most” developers or are still far worse than competent professionals.
  • Cost and token limits are practical constraints; some report rolling back aggressive agentic setups due to expense and slop.
  • Overall sentiment: measuring sloppiness is valuable, but defining and optimizing for true code quality remains unresolved.