Cache is King: A guide for Docker layer caching in GitHub Actions

Original Article ↗ Hacker News Discussion ↗

Engineers trying to speed up Docker builds on GitHub Actions report that most layer-caching strategies over the network negate their benefits, with cache upload/download times often rivaling full rebuilds. Many argue that the only reliable way to get fast, incremental builds is to use self-hosted runners or specialized services that provide large, “sticky” local disks or persistent NVMe-backed caches close to the compute. Alternatives such as Nix, Bazel, Earthly, and external buildkit servers are also raised, but several voices caution that these can add significant complexity and maintenance overhead compared to simpler, disk-based caching setups.

Pain points with Docker layer caching in GitHub Actions

  • Many report that network transfer time for cache save/restore often cancels out build-time gains.
  • Registry-based caches (including “registry mirror” or OCI-based caches) frequently end up no faster than rebuilding, especially for intermediate layers.
  • Only successful builds typically get cached, reducing usefulness for failing or experimental branches.

Local disks, self‑hosted runners, and “sticky disks”

  • Strong consensus that fast, local, persistent storage (NVMe, large disks) is the only consistently effective way to speed Docker builds.
  • “Sticky disks” between runs can dramatically reduce repeated downloads/compiles, but require idempotent steps and careful cleanup.
  • Self-hosted or Kubernetes-based runners with big local disks are seen as the pragmatic solution, though they add infra complexity and isolation concerns in multi-tenant environments.

Third‑party and alternative tools

  • Several services aim to provide faster runners and colocated cache, or persistent BuildKit state.
  • Some use external runners that support custom images and S3-based caches.
  • Others use separate build systems (e.g., Jenkins) mainly to get on-disk Docker layer reuse.

Build system design and alternatives to Dockerfile

  • Some argue the best “solution” is not to use docker build at all: use Bazel + OCI rules, Gradle Jib, or Nix-based image builders for more granular, reliable caching.
  • There is pushback that tools like Bazel and Nix are complex, risky without multiple in-house experts, and can become serious tech debt if “the expert” leaves.
  • Several advocate building artifacts outside Docker and then copying them into minimal images.

GitHub Actions and platform incentives

  • GitHub Actions is praised for easy entry but criticized as hard to scale and awkward to configure for advanced needs (like robust caching).
  • Some suggest GitHub has little incentive to make builds faster since billing is per-minute and focus has shifted toward AI features.
  • Comparisons note other CI systems (GitLab, Bitbucket, CircleCI, Jenkins) having more mature or simpler caching and runner models.

Technical tips, optimizations, and skepticism

  • Tips: use multiple cache-from/cache-to endpoints; prefer local GHA cache; use BuildKit options like image-manifest=true when caching to the same registry; use cache mounts (RUN --mount type=cache).
  • Shrinking image size (removing unused toolchains, libraries) can meaningfully speed builds and pulls.
  • Some contributors conclude that, beyond select cases (e.g., Node builds), complex caching setups are often not worth the engineering effort relative to simpler, uncached builds or better hardware.