A data corruption bug in OpenZFS?

Original Article ↗ Hacker News Discussion ↗

A rare race condition in OpenZFS can cause files to be silently filled with zeros during certain copy operations, without traditional corruption being detectable by scrubs, raising concerns for users who rely on ZFS for data integrity. Commenters emphasize that the bug is hard to trigger and likely hasn’t affected most systems, but use it as a springboard to stress robust, tested backup strategies, selective offsite storage, and awareness of how features like sparse files and snapshots interact with real workloads. The thread also touches on alternatives such as bcachefs, the limitations of Btrfs RAID modes, and the impact of licensing and Oracle’s stewardship on ZFS adoption in Linux.

Impact of the OpenZFS bug

  • Described as a very rare race condition, hard to trigger in practice.
  • Symptoms would be obvious in heavy write/move workloads (e.g., build trees suddenly containing zero‑filled files).
  • Checking bcloneused/bclonesaved is called out as not a reliable test; bclones were only one trigger.
  • Corruption isn’t traditional on‑disk damage: cp reads zeroes and writes them, and ZFS stores them correctly, so scrubs and byte‑level compares with backups won’t detect it.
  • Given the bug’s age and lack of reports from large ZFS users, several comments argue the real‑world risk is low.

Backup reliability and testing

  • Strong emphasis that backups frequently and silently fail: tools may skip files, stop mid‑run, or log poorly.
  • Advice:
    • Test restores regularly, including from systems you rarely touch.
    • Have someone other than the expert perform restores, to validate docs and tooling.
    • Keep scripts and repair tools up to date.
    • Verify backups post‑run (sizes, ability to decrypt/untar, hashes when possible).
    • Use at least two different backup implementations.
  • Database backups are highlighted as especially tricky; filesystem snapshots alone are insufficient for transactional consistency.

Home NAS and large‑data backup strategies

  • Many argue “average home NAS” doesn’t mean 40 TB; for most, only a subset (photos, documents) needs cloud backup.
  • Strategies mentioned:
    • Cloud (Backblaze, S3 Glacier/Deep Archive, Hetzner Storagebox, B2) with tiering by importance.
    • Second NAS or server offsite (family/friends) using rsync/ZFS send/syncthing over VPN/Tailscale/Nebula.
    • Cold storage on external HDD/SSD, rotated and mostly offline.
    • LTO tape for large, critical datasets, with some describing it as cost‑effective long term but operationally tedious and noisy.
  • Rules of thumb: invest ~3× your “hot” storage cost into backups; prioritize multiple non‑redundant copies over a single RAIDed copy.

Sparse files and filesystem abstractions

  • Debate over whether sparse files and SEEK_HOLE/SEEK_DATA were a design mistake or useful optimization.
  • Some see sparse files as leaking block‑level details into user space and adding complexity that can cause bugs like this.
  • Others argue they’re essential for workloads like torrents, databases, VM images, and deduplication.
  • Discussion around fallocate, mmap, and how different filesystems and COW semantics interact with preallocation and fragmentation.

Alternative filesystems and licensing politics

  • Noted that recent showstopper bugs were in OpenZFS, not Oracle’s closed ZFS, but Oracle effectively abandoned Solaris.
  • Long thread on CDDL vs GPL: incompatibility complicates merging OpenZFS into Linux; some argue the legal risk is overstated, others see Oracle as a serious threat.
  • Ubuntu’s long‑term shipping of ZFS without legal action is cited, but some think future lawsuits remain possible.
  • bcachefs is viewed by some as a promising Linux‑native alternative (heterogeneous disks, RAID‑5/6‑style redundancy, SMR‑aware), but others caution it’s still young with limited real‑world exposure and corruption reports.