Why Oxide Chose Illumos

Original Article ↗ Hacker News Discussion ↗

RFD process and transparency

  • Commenters like Oxide’s open RFDs and podcasts explaining decisions.
  • Some see value in formally documenting “obvious” choices; others think this particular RFD reads like retroactive justification.
  • Oxide staff in-thread emphasize: RFDs are internal decision records, later published, not marketing pieces.

Why Illumos as host OS

  • Illumos is viewed as a capable, coherent UNIX with strong primitives (zones, ZFS, SMF) and a culture Oxide trusts.
  • A major advantage cited is ease of landing kernel changes relative to Linux.
  • Several point out that many Oxide engineers previously worked on Solaris/illumos; critics argue this familiarity is the real driver. Oxide engineers reply that they genuinely believe it’s the best trade‑off for their use.

Linux, systemd, and ecosystem concerns

  • The RFD mentions “technological and political” issues around Linux/systemd.
  • Some say this is under‑specified and feels like FUD; others cite specific examples of contentious systemd maintainer behavior and dislike the “sprawl.”
  • Counterpoints: modern Linux distros routinely mix and match non‑systemd components; many enterprises are fine with systemd in practice.

Hypervisor choice: bhyve vs KVM/QEMU/Xen/vmm

  • Oxide chose bhyve on Illumos. There is history of KVM on Illumos and KVM/QEMU at prior companies.
  • A KVM/QEMU developer contests the RFD’s claim that QEMU is “often” unreliable/insecure, citing a relatively good recent vuln record and strong fuzzing; others counter with general C/C++ memory‑safety concerns and Oxide’s Rust‑first philosophy.
  • Trade‑offs discussed: KVM/QEMU feature richness vs complexity and maintenance burden; Xen architecture and dom0 size; OpenBSD vmm’s security‑oriented design but limited features (e.g., historically single vCPU).
  • Nested virtualization is noted as complex but not necessarily slow if tuned.

Service management: SMF vs systemd

  • Some praise illumos SMF as conceptually superior and less intrusive than systemd, though older (XML) and CDDL‑licensed.
  • Others argue systemd has long since surpassed SMF in capabilities and is the de facto standard.

Storage: ZFS, Ceph, and Crucible

  • ZFS on Illumos is seen as a natural fit for a fixed hardware appliance.
  • Ceph is criticized by some as operationally heavy and fragile at scale; others defend it with large real‑world deployments and small ops teams.
  • Oxide’s internal Crucible layer is mentioned as their actual storage service, making any ZFS vs Ceph comparison somewhat indirect.

Market and hardware notes

  • Lack of arm64 support in Illumos is a blocker for some; others note arm64 is still early in “traditional” production.
  • Some think Oxide should target VMware migration opportunities; others say Oxide’s fully integrated rack appliance is a much bigger change than just swapping hypervisors.