Diskless Linux boot using ZFS, iSCSI and PXE

Original Article ↗ Hacker News Discussion ↗

Netbooting Linux from network storage using PXE, iSCSI, and ZFS is highlighted as a flexible way to run “diskless” systems, avoiding local disks while still backing machines with powerful centralized storage. Commenters compare iSCSI to alternatives like NFS and NBD, weighing reliability, performance, and complexity, especially on congested networks or when aiming for high-speed setups (10GbE, RDMA, NVMe‑oF). The thread also branches into practical concerns such as bootloader choices (GRUB vs systemd‑boot vs rEFInd), dual‑booting with Windows, and how similar architectures have been used for thin clients, clusters, and home labs.

NFS vs iSCSI / NBD for Diskless Linux

  • Several commenters note NFS-root is more common and easier to set up, but has caveats:
    • Some package managers (e.g., on Red Hat/Arch) misbehave on NFS roots, possibly due to SQLite.
    • Often used with a read‑only shared root and per‑host writable overlays (ramdisk or per‑host NFS).
  • iSCSI is praised for:
    • Being a standard block protocol, widely supported across OSes (including Windows).
    • Allowing block-level features like disk encryption, volume management, and arbitrary filesystems.
  • NBD is described as:
    • Historically weaker with network interruptions but significantly improved in recent years (pipelining, trim, multiple connections, proper spec and tooling).
    • Still poorly supported for diskless booting in some initramfs tooling (e.g., dracut).
  • One commenter expects iSCSI performance may be worse than NFS; others see value primarily in flexibility, not speed.
  • Some recommend considering NVMe over TCP as a more efficient network storage protocol, especially for fast NVMe drives.

Performance, Networking, and Hardware

  • Multiple people suggest 10GbE as a practical baseline for diskless setups; 1GbE is considered painful.
  • Even on 10GbE, remote iSCSI is expected to be significantly slower than local NVMe, but still usable.
  • iSCSI is said to behave poorly on congested networks or with incast traffic:
    • Recommendations include QoS prioritization, dedicated VLANs or separate “storage networks,” and switches with large buffers.
    • Hardware offload via specialized NICs (Chelsio, Mellanox/RDMA) can substantially improve performance.
  • NVMe‑oF is described as offering near‑local latency/throughput when properly set up.

Boot Mechanisms: PXE, initrd, and “Diskless” Semantics

  • Clarification that PXE can load kernel+initramfs entirely into RAM; no local disk is needed.
  • Some suggest booting into a minimal network-only initrd that then downloads another kernel and kexecs into it.
  • Debate around whether “diskless” is accurate when a remote disk (iSCSI/NFS/NBD) is used; consensus is it’s diskless from the client’s hardware perspective.

Bootloaders, UEFI, and Secure Boot

  • Long subthread on bootloaders:
    • Strong criticism of GRUB’s complexity and UX; some say most failure modes vanish if you avoid it.
    • Alternatives promoted: systemd‑boot, rEFInd, EFI stub boot, Unified Kernel Images (UKI), ZFSBootMenu.
    • Some argue GRUB remains valuable because it runs on BIOS and many architectures and OSes.
  • Disagreement on how hard UEFI management really is:
    • Some say you don’t need to edit UEFI entries often; hooks, fixed filenames (e.g., /vmlinuz), or UKIs can simplify updates.
    • Others report repeated problems with Windows updates purging non‑Windows boot entries.
  • Secure Boot:
    • Some call it a hassle; others say modern distros mostly “just work” with pre-signed bootloaders, with a few hardware/vendor exceptions.
    • rEFInd and systemd‑boot are reported to work with Secure Boot; measured boot support is mentioned for systemd‑boot.

ZFS and Data Integrity

  • ZFS is largely used here on the server side to store images/zvols backing network roots.
  • There’s a question whether ZFS can correct errors or only detect them:
    • Responses state ZFS does self‑healing: on checksum errors, bad copies are repaired from good ones if redundancy exists, either during normal reads or via scrubs.
    • Metadata is reported to have multiple copies by default, even on single disks.

Alternative Systems and Simpler Setups

  • Several prior art references:
    • LTSP-style diskless Linux offices with clients running from a single server image in RAM.
    • Diskless virtual machines with Xen/KVM.
    • Tools like iVentoy for easy PXE boot menus.
  • One commenter contrasts the complexity (Ansible, Apache, etc.) with Plan 9 / 9front netboot, which they claim is much simpler due to integrated DHCP/TFTP and configuration.
  • Nostalgic mentions of Open Firmware’s boot net and older BIOS-era workflows.
  • General theme: the showcased stack is one of many; some see it as a fun learning project, others as overcomplicated compared to their preferred ecosystems and tools.