Learnings from our years of Kubernetes in production

Original Article ↗ Hacker News Discussion ↗

Kubernetes in production draws mixed reactions, with many engineers arguing its complexity, operational pitfalls (like certificate expiry taking down clusters), and constant upgrade burden are unjustified for small teams or simple apps that could run on VMs or lighter-weight container platforms. Others highlight benefits such as a unified abstraction for deployment, portability between on‑prem and cloud, and rich tooling—provided you use managed services (EKS/AKS/GKE), keep clusters non‑snowflake, and invest in a platform team. The thread also touches on alternatives like ECS, Nomad, Ray, and Proxmox, trade‑offs around stateful workloads and dev environments, and a broader concern that Kubernetes is often adopted early for perceived scalability rather than proven need.

Kubernetes for small teams vs. large orgs

  • Many argue Kubernetes is overkill for small teams/startups; complexity and ownership cost crush velocity.
  • Some say it makes sense mainly when you have hundreds of engineers and a dedicated platform team.
  • Others counter that team size is less important than problem complexity and scale; some individuals even use k8s for personal projects because they like having one standard abstraction.

Managed vs self‑managed clusters

  • Strong consensus: self‑managing control planes is risky; cert expiry and upgrade issues have caused multi‑day outages.
  • Managed offerings (EKS/AKS/GKE) are seen as far more reliable and dramatically reduce pain, though not outage‑free.
  • Several commenters view the article’s outages as more about poor operational practices (no manifests/backup, bad cert management) than inherent k8s flaws.

Complexity, abstractions, and benefits

  • Critics list steep learning curves: PKI/certs, etcd, CNI networking, DNS, ingress controllers, Helm, GitOps, YAML sprawl, frequent deprecations.
  • Supporters highlight the unified resource model: same manifests across on‑prem/cloud, pluggable storage/ingress, automatic service discovery, scaling, failover, and node maintenance workflows.
  • Some describe k8s as an excellent dev/QA platform (consistent environments, easy to spin up complex topologies) but “just ok” for production compared to more opinionated runtimes.

Alternatives and simpler approaches

  • Suggested options: ECS, Fargate, Azure Container Apps, Nomad, Ray, Docker Compose/Swarm, Proxmox, Ubuntu MicroCloud, plain VMs with CI/CD, or monoliths on autoscaled servers.
  • For many products, commenters believe a few VMs + load balancer + managed DB would have been cheaper, simpler, and sufficiently scalable.

Developer environments & GitOps

  • Some teams run local/dev clusters (kind/k3s/k3d, Tilt, kustomize) or small shared clusters mirroring prod, often with mocks for S3 and other services.
  • GitOps (Argo/Flux) gets mixed reactions: some see it as essential for versioning and drift detection; others think it adds another sync layer and prefer direct Helm/kubectl.

Cost, staffing, and TCO

  • Frequent concern: articles rarely quantify infra costs, engineer time, or opportunity cost vs. simpler stacks.
  • Several note needing at least partial FTEs to own k8s even on managed services.

“Learnings” vs “lessons”

  • Long sub‑thread debates “learnings” as corporate jargon vs. acceptable modern usage; many find it grating, some defend it as a useful nuance.