A skeptic's first contact with Kubernetes

Original Article ↗ Hacker News Discussion ↗

Kubernetes is praised for its control-loop architecture and extensibility, but many engineers argue that in practice it has become overly complex, especially around configuration management with YAML and Helm. Commenters highlight how core features like networking, storage, autoscaling, and cloud integration are increasingly pushed into pluggable components, which enables powerful customization but also means every cluster ends up uniquely assembled and harder to reason about. Several alternatives and tooling approaches are mentioned—from kustomize, CUE, and Jsonnet to Terraform-like and “real language” generators—reflecting an ongoing search for a cleaner, more ergonomic way to deploy and scale containerized applications.

Vendor integrations, CSI/CNI, and extensibility

  • Kubernetes is steadily removing in-tree, vendor-specific code (cloud providers, storage, etc.) in favor of out-of-tree components via CSI, CNI, and operators.
  • Core idea: Kubernetes is a generic control-plane and control-loop framework; almost every subsystem (scheduler, kubelet, networking, autoscaling) can be swapped out.
  • This extensibility is seen both as a strength (adaptable to many environments) and a cause of confusion and fragmentation.

Helm, YAML, and configuration tooling

  • Strong dissatisfaction with Helm’s text-based YAML templating; failures can leave partial resources that must be cleaned up manually.
  • Helm remains dominant mainly because of its massive chart ecosystem and packaging conventions, not because the templating model is liked.
  • Many alternatives are mentioned: Kustomize, Jsonnet/Ksonnet, CUE, Dhall, RCL, KCL, CDK8s, Terraform’s HCL, Ruby/TypeScript/Python generators.
  • Ongoing debate:
    • One camp wants “real” programming languages (Python, Ruby, TypeScript, Starlark) to generate manifests.
    • Another prefers constrained config/templating languages for safety, despite tooling pain.
  • Some report success with GitOps flows (e.g., Flux + raw YAML + Kustomize) and strict separation between “generate manifests” and “deploy manifests.”

Kustomize in kubectl

  • Kustomize is popular for simpler setups and is currently embedded in kubectl.
  • There is a proposal to remove the embedded version (and use it as an external tool instead), but the outcome is unclear due to compatibility concerns.

Autoscaling, metrics, and observability

  • Horizontal Pod Autoscaler can already use custom or external metrics (e.g., queue depth) via metrics adapters.
  • KEDA extends this model with rich triggers (e.g., Prometheus metrics, database-backed queue depth).
  • Karpenter provides advanced node autoscaling, especially for cloud “spot” fleets.
  • Kubernetes exposes extensive metrics (kubelet, controllers, kube-state-metrics), enabling alerts on failed reconciliation and unstable states.

Complexity, suitability, and break-even

  • Some argue Kubernetes is appropriate for “complex-domain” infrastructure (Cynefin), unknown or highly variable machine counts, or many heterogeneous tech stacks.
  • Others see it frequently misapplied, noting projects that ran worse on Kubernetes and preferring simpler container hosts or a small number of well-managed servers.
  • Suggested “break-even” heuristics: when you don’t know how many machines you’ll need, or when you have multiple distinct stacks that are painful to manage with ad-hoc tools.

Networking model and IPv6

  • The standard pod/service networking abstraction is criticized by some as replicating “hard exterior, soft interior” corporate networks.
  • One viewpoint favors globally routable IPv6 for all workloads with security handled at higher layers; others find that idea unsettling or unnecessary.
  • Discussion touches on overlays vs L3/BGP-based CNIs; what counts as an “overlay” is contested.

Cluster uniqueness and distributions

  • Pushing more functionality out-of-tree implies every cluster ends up unique in its exact mix of CNIs, CSIs, and controllers.
  • Some see this as inevitable and argue that curated distributions and managed services exist precisely to standardize sane combinations and reduce operational burden.