Cache is King: A guide for Docker layer caching in GitHub Actions

Original Article ↗ Hacker News Discussion ↗

Pain points with Docker layer caching in GitHub Actions

  • Many report that network transfer time for cache save/restore often cancels out build-time gains.
  • Registry-based caches (including “registry mirror” or OCI-based caches) frequently end up no faster than rebuilding, especially for intermediate layers.
  • Only successful builds typically get cached, reducing usefulness for failing or experimental branches.

Local disks, self‑hosted runners, and “sticky disks”

  • Strong consensus that fast, local, persistent storage (NVMe, large disks) is the only consistently effective way to speed Docker builds.
  • “Sticky disks” between runs can dramatically reduce repeated downloads/compiles, but require idempotent steps and careful cleanup.
  • Self-hosted or Kubernetes-based runners with big local disks are seen as the pragmatic solution, though they add infra complexity and isolation concerns in multi-tenant environments.

Third‑party and alternative tools

  • Several services aim to provide faster runners and colocated cache, or persistent BuildKit state.
  • Some use external runners that support custom images and S3-based caches.
  • Others use separate build systems (e.g., Jenkins) mainly to get on-disk Docker layer reuse.

Build system design and alternatives to Dockerfile

  • Some argue the best “solution” is not to use docker build at all: use Bazel + OCI rules, Gradle Jib, or Nix-based image builders for more granular, reliable caching.
  • There is pushback that tools like Bazel and Nix are complex, risky without multiple in-house experts, and can become serious tech debt if “the expert” leaves.
  • Several advocate building artifacts outside Docker and then copying them into minimal images.

GitHub Actions and platform incentives

  • GitHub Actions is praised for easy entry but criticized as hard to scale and awkward to configure for advanced needs (like robust caching).
  • Some suggest GitHub has little incentive to make builds faster since billing is per-minute and focus has shifted toward AI features.
  • Comparisons note other CI systems (GitLab, Bitbucket, CircleCI, Jenkins) having more mature or simpler caching and runner models.

Technical tips, optimizations, and skepticism

  • Tips: use multiple cache-from/cache-to endpoints; prefer local GHA cache; use BuildKit options like image-manifest=true when caching to the same registry; use cache mounts (RUN --mount type=cache).
  • Shrinking image size (removing unused toolchains, libraries) can meaningfully speed builds and pulls.
  • Some contributors conclude that, beyond select cases (e.g., Node builds), complex caching setups are often not worth the engineering effort relative to simpler, uncached builds or better hardware.