Why Oxide Chose Illumos
Oxide’s choice to build its cloud appliance on Illumos and the bhyve hypervisor, rather than Linux with KVM/QEMU or other hypervisors like Xen, prompts debate over technical trade-offs, team familiarity, and long‑term maintainability. Commenters weigh Illumos features such as SMF and integrated ZFS against Linux’s broader ecosystem and systemd’s dominance, and contrast bhyve’s design and complexity with more widely deployed options. Many conclude that, for a tightly integrated, hardware-specific platform run by former Sun/Joyent engineers, optimizing for in‑house expertise and an opinionated, controllable stack can be more important than following the industry’s most popular choices.
RFD process and transparency
- Commenters like Oxide’s open RFDs and podcasts explaining decisions.
- Some see value in formally documenting “obvious” choices; others think this particular RFD reads like retroactive justification.
- Oxide staff in-thread emphasize: RFDs are internal decision records, later published, not marketing pieces.
Why Illumos as host OS
- Illumos is viewed as a capable, coherent UNIX with strong primitives (zones, ZFS, SMF) and a culture Oxide trusts.
- A major advantage cited is ease of landing kernel changes relative to Linux.
- Several point out that many Oxide engineers previously worked on Solaris/illumos; critics argue this familiarity is the real driver. Oxide engineers reply that they genuinely believe it’s the best trade‑off for their use.
Linux, systemd, and ecosystem concerns
- The RFD mentions “technological and political” issues around Linux/systemd.
- Some say this is under‑specified and feels like FUD; others cite specific examples of contentious systemd maintainer behavior and dislike the “sprawl.”
- Counterpoints: modern Linux distros routinely mix and match non‑systemd components; many enterprises are fine with systemd in practice.
Hypervisor choice: bhyve vs KVM/QEMU/Xen/vmm
- Oxide chose bhyve on Illumos. There is history of KVM on Illumos and KVM/QEMU at prior companies.
- A KVM/QEMU developer contests the RFD’s claim that QEMU is “often” unreliable/insecure, citing a relatively good recent vuln record and strong fuzzing; others counter with general C/C++ memory‑safety concerns and Oxide’s Rust‑first philosophy.
- Trade‑offs discussed: KVM/QEMU feature richness vs complexity and maintenance burden; Xen architecture and dom0 size; OpenBSD vmm’s security‑oriented design but limited features (e.g., historically single vCPU).
- Nested virtualization is noted as complex but not necessarily slow if tuned.
Service management: SMF vs systemd
- Some praise illumos SMF as conceptually superior and less intrusive than systemd, though older (XML) and CDDL‑licensed.
- Others argue systemd has long since surpassed SMF in capabilities and is the de facto standard.
Storage: ZFS, Ceph, and Crucible
- ZFS on Illumos is seen as a natural fit for a fixed hardware appliance.
- Ceph is criticized by some as operationally heavy and fragile at scale; others defend it with large real‑world deployments and small ops teams.
- Oxide’s internal Crucible layer is mentioned as their actual storage service, making any ZFS vs Ceph comparison somewhat indirect.
Market and hardware notes
- Lack of arm64 support in Illumos is a blocker for some; others note arm64 is still early in “traditional” production.
- Some think Oxide should target VMware migration opportunities; others say Oxide’s fully integrated rack appliance is a much bigger change than just swapping hypervisors.