We're Leaving Kubernetes

Original Article ↗ Hacker News Discussion ↗

Scope and Use Case

  • Several commenters stress the article is about cloud-based development environments, not production apps.
  • Gitpod’s workload: long‑lived, stateful, multi‑tenant dev environments, often with GPUs, backing a commercial product and self‑hosted installs.
  • Some readers initially confuse this with “internal dev envs” and call the use case niche; others argue it’s representative of ML/data‑heavy teams.

Why Kubernetes Struggled for This

  • Kubernetes is seen as optimized for relatively homogeneous, stateless, pre‑configured services, not per‑user, highly dynamic dev workspaces.
  • State and storage are recurring pain points: PVC variability across clusters, unreliable local disks, expensive backups, slow image pulls and restores.
  • Security/isolation gaps: user namespaces have improved but issues around /proc, FUSE, and running untrusted or near‑root workloads remain.
  • Debugging a dev env inside Kubernetes is described as a complexity cliff; self‑hosted customers see widely varying k8s setups.
  • Some argue these are intrinsic mismatches; others claim they’re solvable but the ROI is poor.

Gitpod Flex and New Architecture (Unclear Details)

  • Gitpod staff mention a new runner‑based architecture (“Gitpod Flex”) using AWS primitives (EC2, EBS, ECS) and a generic runner interface that can also support local/desktop.
  • Many readers are dissatisfied that the article doesn’t technically detail Flex; follow‑up posts are promised.

Reactions to Leaving Kubernetes

  • Strong agreement from many who found k8s a costly distraction for small/medium setups, especially for GPU/AI and dev workloads.
  • Counter‑view: k8s is still excellent for stable, stateless production; some say Gitpod could have kept k8s as a base layer or per‑customer clusters.
  • A few call the decision a management/architecture failure rather than an indictment of Kubernetes itself.

Alternatives and Patterns Discussed

  • Serverless/container PaaS (e.g., Cloud Run, AppRunner‑like services) praised for cost, simplicity, fast autoscaling, and “Kubernetes as someone else’s problem.”
  • VM‑heavy or microVM approaches (Firecracker, Kata, LXD/Incus, Nomad, plain VMs) suggested for isolation and predictable performance.
  • For dev envs: local laptops with containers/VMs, docker‑compose, devcontainers, distrobox/toolbx, and Nix/Guix ecosystems are all advocated.

Remote vs Local Dev Environments

  • Large debate:
    • Pro‑local: best latency and feedback loop, easier debugging, fewer moving parts; “just buy beefy laptops and run things locally.”
    • Pro‑remote: necessary when stacks, datasets, or GPU needs exceed laptops; improves security and standardization; but requires serious investment and org discipline.
  • Consensus: remote dev can work well at large scale, but is easy to get wrong and often feels worse than local if under‑engineered.