Diskless Linux boot using ZFS, iSCSI and PXE

Original Article ↗ Hacker News Discussion ↗

NFS vs iSCSI / NBD for Diskless Linux

  • Several commenters note NFS-root is more common and easier to set up, but has caveats:
    • Some package managers (e.g., on Red Hat/Arch) misbehave on NFS roots, possibly due to SQLite.
    • Often used with a read‑only shared root and per‑host writable overlays (ramdisk or per‑host NFS).
  • iSCSI is praised for:
    • Being a standard block protocol, widely supported across OSes (including Windows).
    • Allowing block-level features like disk encryption, volume management, and arbitrary filesystems.
  • NBD is described as:
    • Historically weaker with network interruptions but significantly improved in recent years (pipelining, trim, multiple connections, proper spec and tooling).
    • Still poorly supported for diskless booting in some initramfs tooling (e.g., dracut).
  • One commenter expects iSCSI performance may be worse than NFS; others see value primarily in flexibility, not speed.
  • Some recommend considering NVMe over TCP as a more efficient network storage protocol, especially for fast NVMe drives.

Performance, Networking, and Hardware

  • Multiple people suggest 10GbE as a practical baseline for diskless setups; 1GbE is considered painful.
  • Even on 10GbE, remote iSCSI is expected to be significantly slower than local NVMe, but still usable.
  • iSCSI is said to behave poorly on congested networks or with incast traffic:
    • Recommendations include QoS prioritization, dedicated VLANs or separate “storage networks,” and switches with large buffers.
    • Hardware offload via specialized NICs (Chelsio, Mellanox/RDMA) can substantially improve performance.
  • NVMe‑oF is described as offering near‑local latency/throughput when properly set up.

Boot Mechanisms: PXE, initrd, and “Diskless” Semantics

  • Clarification that PXE can load kernel+initramfs entirely into RAM; no local disk is needed.
  • Some suggest booting into a minimal network-only initrd that then downloads another kernel and kexecs into it.
  • Debate around whether “diskless” is accurate when a remote disk (iSCSI/NFS/NBD) is used; consensus is it’s diskless from the client’s hardware perspective.

Bootloaders, UEFI, and Secure Boot

  • Long subthread on bootloaders:
    • Strong criticism of GRUB’s complexity and UX; some say most failure modes vanish if you avoid it.
    • Alternatives promoted: systemd‑boot, rEFInd, EFI stub boot, Unified Kernel Images (UKI), ZFSBootMenu.
    • Some argue GRUB remains valuable because it runs on BIOS and many architectures and OSes.
  • Disagreement on how hard UEFI management really is:
    • Some say you don’t need to edit UEFI entries often; hooks, fixed filenames (e.g., /vmlinuz), or UKIs can simplify updates.
    • Others report repeated problems with Windows updates purging non‑Windows boot entries.
  • Secure Boot:
    • Some call it a hassle; others say modern distros mostly “just work” with pre-signed bootloaders, with a few hardware/vendor exceptions.
    • rEFInd and systemd‑boot are reported to work with Secure Boot; measured boot support is mentioned for systemd‑boot.

ZFS and Data Integrity

  • ZFS is largely used here on the server side to store images/zvols backing network roots.
  • There’s a question whether ZFS can correct errors or only detect them:
    • Responses state ZFS does self‑healing: on checksum errors, bad copies are repaired from good ones if redundancy exists, either during normal reads or via scrubs.
    • Metadata is reported to have multiple copies by default, even on single disks.

Alternative Systems and Simpler Setups

  • Several prior art references:
    • LTSP-style diskless Linux offices with clients running from a single server image in RAM.
    • Diskless virtual machines with Xen/KVM.
    • Tools like iVentoy for easy PXE boot menus.
  • One commenter contrasts the complexity (Ansible, Apache, etc.) with Plan 9 / 9front netboot, which they claim is much simpler due to integrated DHCP/TFTP and configuration.
  • Nostalgic mentions of Open Firmware’s boot net and older BIOS-era workflows.
  • General theme: the showcased stack is one of many; some see it as a fun learning project, others as overcomplicated compared to their preferred ecosystems and tools.