A skeptic's first contact with Kubernetes

Original Article ↗ Hacker News Discussion ↗

Vendor integrations, CSI/CNI, and extensibility

  • Kubernetes is steadily removing in-tree, vendor-specific code (cloud providers, storage, etc.) in favor of out-of-tree components via CSI, CNI, and operators.
  • Core idea: Kubernetes is a generic control-plane and control-loop framework; almost every subsystem (scheduler, kubelet, networking, autoscaling) can be swapped out.
  • This extensibility is seen both as a strength (adaptable to many environments) and a cause of confusion and fragmentation.

Helm, YAML, and configuration tooling

  • Strong dissatisfaction with Helm’s text-based YAML templating; failures can leave partial resources that must be cleaned up manually.
  • Helm remains dominant mainly because of its massive chart ecosystem and packaging conventions, not because the templating model is liked.
  • Many alternatives are mentioned: Kustomize, Jsonnet/Ksonnet, CUE, Dhall, RCL, KCL, CDK8s, Terraform’s HCL, Ruby/TypeScript/Python generators.
  • Ongoing debate:
    • One camp wants “real” programming languages (Python, Ruby, TypeScript, Starlark) to generate manifests.
    • Another prefers constrained config/templating languages for safety, despite tooling pain.
  • Some report success with GitOps flows (e.g., Flux + raw YAML + Kustomize) and strict separation between “generate manifests” and “deploy manifests.”

Kustomize in kubectl

  • Kustomize is popular for simpler setups and is currently embedded in kubectl.
  • There is a proposal to remove the embedded version (and use it as an external tool instead), but the outcome is unclear due to compatibility concerns.

Autoscaling, metrics, and observability

  • Horizontal Pod Autoscaler can already use custom or external metrics (e.g., queue depth) via metrics adapters.
  • KEDA extends this model with rich triggers (e.g., Prometheus metrics, database-backed queue depth).
  • Karpenter provides advanced node autoscaling, especially for cloud “spot” fleets.
  • Kubernetes exposes extensive metrics (kubelet, controllers, kube-state-metrics), enabling alerts on failed reconciliation and unstable states.

Complexity, suitability, and break-even

  • Some argue Kubernetes is appropriate for “complex-domain” infrastructure (Cynefin), unknown or highly variable machine counts, or many heterogeneous tech stacks.
  • Others see it frequently misapplied, noting projects that ran worse on Kubernetes and preferring simpler container hosts or a small number of well-managed servers.
  • Suggested “break-even” heuristics: when you don’t know how many machines you’ll need, or when you have multiple distinct stacks that are painful to manage with ad-hoc tools.

Networking model and IPv6

  • The standard pod/service networking abstraction is criticized by some as replicating “hard exterior, soft interior” corporate networks.
  • One viewpoint favors globally routable IPv6 for all workloads with security handled at higher layers; others find that idea unsettling or unnecessary.
  • Discussion touches on overlays vs L3/BGP-based CNIs; what counts as an “overlay” is contested.

Cluster uniqueness and distributions

  • Pushing more functionality out-of-tree implies every cluster ends up unique in its exact mix of CNIs, CSIs, and controllers.
  • Some see this as inevitable and argue that curated distributions and managed services exist precisely to standardize sane combinations and reduce operational burden.