Unfashionably secure: why we use isolated VMs
Virtual machines are seeing renewed favor as some engineers push back against the dominance of Docker and Kubernetes, arguing that per-tenant VMs provide clearer isolation, simpler mental models, and better security guarantees than multi-tenant container clusters. Others counter that containers dramatically improve reproducibility, deployment workflows, and cross-distro packaging, especially when paired with orchestration tools, even if those tools are often overapplied to smaller-scale systems. Threaded through the debate are concerns about operational complexity, storage persistence, cloud costs, vendor lock‑in, and the lack of mature, Docker-like tooling for VM- or microVM-based approaches.
VMs vs Containers: Security & Isolation
- Many argue VMs provide a stronger, clearer security boundary than containers/jails/zones because they don’t share a kernel; containers’ entire host kernel remains in the TCB.
- Others note that even VMs are vulnerable to side channels (Spectre, rowhammer, etc.) and that jail/zone/container security differences mostly come from fewer bugs, not fundamentally better designs.
- Some see “one-VM-per-tenant” as a straightforward way to limit blast radius; others call it wasteful compared to well‑designed multi‑tenant setups.
Developer Experience & Reproducibility
- Containers are widely praised for reproducible dev environments and eliminating “works on my machine” issues; many hobbyists and professionals run most services via Docker/docker-compose.
- Critics say containers just shift dependency hell into images and don’t solve versioning/update problems cleanly; Docker is described as a huge lockfile.
- Several treat Docker as a de facto cross‑distro “package + service manager,” while others argue it lacks real dependency metadata and introspection.
Kubernetes and Orchestration Debate
- Strong sentiment that k8s is overused outside “hyper‑scale” scenarios and often adopted for résumé value.
- Complaints: complexity, fragile storage/network integrations (CSI/CNI), awkward handling of stateful workloads and live migration, and an ecosystem of heavyweight add‑ons.
- Supporters say k8s gives a uniform API for deployment, scaling, and service discovery across heterogeneous stacks, which simplifies operations in large organizations.
Cost, Cloud, and Resource Efficiency
- Some claim cloud + containers is vastly more expensive than owning hardware; scaling could be handled with a few powerful servers instead of many instances.
- Others rely on VMs/containers in public clouds for elasticity and to avoid up‑front hardware and ops expertise.
Storage, Persistence, and State
- Several find container storage semantics confusing and fragile; misconfigured volumes can cause silent data loss.
- Read‑only root filesystems are suggested but clash with many existing apps’ expectations.
Alternatives and Hybrids
- Discussion of BSD jails, Solaris zones, Nix/Guix, OStree, microVMs (Firecracker/Kata), gVisor, unikernels, and Qubes‑style compartmentalization.
- Some run containers inside VMs, or one container/VM per tenant, seeking a balance between DX and isolation.