AI coding has made CI a bottleneck, so we reworked ours to keep up

Original Article ↗ Hacker News Discussion ↗

AI-powered coding is massively increasing the volume and speed of code changes, exposing continuous integration (CI) pipelines as a new bottleneck. Developers are experimenting with faster build systems (like Bazel), self-hosted or third‑party runners, aggressive caching, and even shifting more checks to local or agent loops to keep feedback times acceptable as test suites balloon in size. Many commenters also question how much of this extra testing actually improves product quality, suggesting that human QA, product decisions, and user value remain the real constraints even as raw coding and CI throughput accelerate.

CI as the New Bottleneck

  • Many report CI pipelines becoming the limiting factor once AI agents massively increase code/PR volume.
  • Long builds (10+ minutes) slow both humans and agents; people aim for sub‑2–5 minute cycles, sometimes under 60 seconds.
  • Some argue the real bottleneck is now QA, product validation, and customers (adoption, update fatigue), not just CI.

Bazel, Caching, and Build Systems

  • Bazel is praised for ~10s incremental builds on large polyglot codebases with a warm cache; agents make Bazel migrations far faster than before.
  • Others highlight tradeoffs: complex setup, per-task sandbox overhead, ergonomics that require parallel “normal” tooling, and mixed results for JS/TS.
  • Remote caching can be limited by network and disk I/O; optimizations include fetching only digests, content-defined chunking for large artifacts, and building container images hermetically (without Dockerfile).
  • Some advocate Bazel-like but simpler systems (e.g., RWX, custom tools) and unified schedulers that treat build + test as one graph.

CI Infrastructure Choices

  • Hosted GitHub Actions is seen as slow/unreliable and expensive at scale; many move to:
    • Third-party runners with faster CPUs/NVMe and better caching.
    • Self‑hosted runners (on‑prem or cheap cloud boxes) with warm workspaces, LXD containers, and aggressive parallelization.
  • Direct peering / network links to GitHub can still be flaky; local caches mitigate but don’t fully solve it.

Agentic Coding, Tests, and QA

  • Agents generate huge numbers of tests; some pipelines report test suites roughly quadrupling.
  • Several commenters see many AI-written unit tests as trivial or redundant “slop”; end‑to‑end and characterization tests are considered more valuable.
  • Some teams now review tests more than code, trusting that if relevant tests pass, large refactors are likely safe.
  • Others question whether 4× more tests yield 4× more value, or just higher CI cost.

Rethinking SDLC and Architecture

  • Suggestions include:
    • Moving most lint/unit checks into the agent loop and keeping CI for heavy/irreplicable checks.
    • Allowing more CI work on developer machines with signed artifacts, though supply‑chain risk is noted.
    • Designing smaller, contract‑oriented services or modular monoliths to let agents iterate faster within bounded contexts.

Do AI Productivity Gains Improve Products?

  • Some report clear wins: big refactors and migrations completed in days instead of months, performance gains, infra cleanups, accessibility improvements, and more ambitious solo/indie projects.
  • Others see mostly more code, more trivial apps, buggier software, and little visible improvement in flagship products or macro productivity, framing much of AI‑accelerated coding as “more code, not more value.”