Go, Containers, and the Linux Scheduler

Original Article ↗ Hacker News Discussion ↗

Containerized Go services often misjudge how much CPU they actually have, leading the Go runtime to spawn too many threads and suffer severe throttling under Linux’s CFS scheduler. Commenters compare CPU quotas vs. shares, argue over whether to rely on Kubernetes/Docker limits or application-level tuning, and point to tools like `automaxprocs` and cgroup-aware runtimes (as in Java and .NET) as practical fixes. More broadly, the exchange highlights the trade-offs between strict CPU limits, overcommit for better utilization, and predictable latency in large multi-tenant clusters.

Go, CFS, and container CPU limits

  • Main issue: Go’s runtime sizes GOMAXPROCS from visible cores, ignoring cgroup CPU quotas, so in containers it often spawns far too many OS threads for the CPU time actually available.
  • This leads to heavy CFS throttling, degraded throughput and latency, and “wasted cores” behavior seen over many years.
  • Some argue the core bug is in Go’s runtime (using the wrong metric), not the Linux scheduler.

Docker/Kubernetes: quotas vs shares and flags

  • --cpus in Docker maps to CFS quota/period, not cpuset, and can cause hard throttling even on mostly idle hosts.
  • cpu.shares is proportional/relative scheduling used only under contention; quotas (cpu.cfs_quota_us) are hard time limits.
  • In Kubernetes: CPU request → shares; CPU limit → quota. Some say “don’t set limits, only requests”; others see quota as essential for strong isolation.
  • There is disagreement about when CFS quota actually throttles: some claim “only under contention”; others point to docs and experiments showing it is a hard cap regardless.

Workarounds: GOMAXPROCS and cgroup probing

  • Popular pattern: set GOMAXPROCS from cgroup data (e.g., cpu.cfs_quota_us / cpu.cfs_period_us) or use libraries like automaxprocs.
  • Several reports of substantial latency and throughput gains after doing this, versus both “no limit” and “with quota but no tuning”.
  • Others warn that capping GOMAXPROCS to quota can hurt burst handling and increase tail latency; “max parallelism should not equal long‑term CPU budget.”
  • Suggestions include: mutating webhooks to inject GOMAXPROCS, using nproc/sched_getaffinity, or lxcfs to fake /proc for container‑aware views.

Debate: should you use CPU limits at all?

  • One camp: “Stop using CPU limits; use only reservations/requests.” Argument: limits waste CPU, cause throttling while host is idle, and make capacity behavior unpredictable.
  • Counter‑camp: limits are needed to:
    • Prevent noisy neighbors from impacting critical services.
    • Avoid relying on “free burst CPU” that may disappear as nodes fill.
    • Simulate worst‑case conditions for realistic capacity planning.
  • Several note that overcommitment and mixing latency‑sensitive vs batch workloads is inherently complex; K8s abstractions can mislead less‑experienced ops teams.

Other runtimes and container awareness

  • Java, .NET, and Rust have added container‑aware logic, using cgroup limits to size thread pools, GC threads, etc.
  • Java’s behavior has evolved from using shares to quotas; there are flags to control whether container quota influences CPU count.
  • Some suggest similar mechanisms or preloadable shims could be applied to Go.

Containers vs VMs/unikernels for Go

  • Recurrent question: since Go produces static binaries, what value do containers add?
  • Pro‑container points: standardized packaging, network/FS/process isolation, resource controls, orchestration compatibility (Kubernetes), and easier CI/CD and multi‑service setups.
  • Skeptical view: for single Go services, containers can be redundant “bloat”; unikernels plus Go binaries may be a better fit, though tooling and deployment are still rough.
  • Consensus: ecosystem and workflow standardization are major reasons Go apps still end up in containers.

Miscellaneous / unclear points

  • Some discussion of Kubernetes CPU manager policies (static vs none) and how they change visible CPU masks; behavior differs across setups.
  • Mention of emerging Linux schedulers (EEVDF) but no concrete production experience reported in the thread.
  • Overall, participants agree the interaction of language runtimes, CFS, cgroups, and orchestrators remains subtle and easy to misconfigure.