Making EC2 boot time faster

Original Article ↗ Hacker News Discussion ↗

Engineers are probing why Amazon EC2 instances can take tens of seconds to minutes to become usable, especially for ephemeral workloads like CI runners, and how to cut that latency. Much of the focus is on EBS behavior—lazy loading root volumes from S3, lack of true copy‑on‑write, and slow snapshot hydration—and workarounds such as booting once then stopping instances, minimizing AMI size, or bypassing EBS entirely with S3- or NVMe-backed root filesystems. Commenters also weigh broader trade-offs: AWS warm pools and fast snapshot restore vs. their cost and lag, alternative platforms using Firecracker or unikernels for millisecond cold starts, and whether simpler on-prem or bare-metal setups would outperform today’s layered cloud abstractions.

Why EC2 boot is slow (EBS, snapshots, hydration)

  • Main bottleneck: creating and “hydrating” an EBS root volume from a snapshot stored in S3.
  • First access to blocks is lazy and slow; later boots on the same volume are much faster.
  • EBS doesn’t support true read‑only or copy‑on‑write roots, nor booting directly from local NVMe, so each instance pays a full-volume cost.
  • Some report strong bi‑modal boot times (tens vs ~80s) with unclear but likely capacity / scheduling causes.

Workarounds and optimizations on AWS

  • Strategy in the article: boot once, stop the instance, then later start it from the already‑hydrated EBS volume.
  • Warm Pools exist but are reported as too slow to react (60s+ to notice scale-up) and have feature limits.
  • Minimizing image size and using sparse EBS volumes can reduce hydration time.
  • Fast Snapshot Restore and EBS “fast restore”/hydration with fio help but are expensive and still slower than desired.
  • Some note Amazon Linux 2023 boots faster than Ubuntu.

Alternative architectures: S3, immutable roots, ephemeral storage

  • Several propose tiny boot AMIs that fetch a squashfs/root image from S3 and run from RAM or overlayfs, using local NVMe for scratch.
  • s3fs is called out as unreliable in production; rclone mount suggested instead.
  • Others argue this essentially re‑implements EBS/AMIs, while proponents say what’s missing is cheap read‑only shared roots with CoW overlays.

CI, GitHub Actions, and latency sources

  • Even with fast EC2 boot, GitHub’s own runner startup and job assignment can add ~8–10s, partly negating gains.
  • Some vendors use scale, warm pools, or alternative stacks (Firecracker, unikernels) to get sub‑second to tens‑of‑ms cold starts for CI‑like workloads.

Autoscaling and prediction window

  • Shorter boot times shrink the prediction horizon and make autoscaling more accurate and cost‑efficient.
  • Debate on typical timescales: many workloads can live with 30–60s; others (spiky CI, interactive environments) need much faster reaction.

Cloud vs bare metal / other providers

  • Some argue a single powerful bare‑metal box (or smaller clouds like Hetzner) can give much faster, simpler builds.
  • Others stress that large clouds optimize for multi‑tenant isolation and networked storage, trading off raw latency for flexibility and durability.