Btrfs/ZFS/bcachefs under workloads classic benchmarks skip

A new benchmark suite comparing modern copy‑on‑write filesystems like ZFS, Btrfs, and bcachefs under multi‑device workloads has drawn interest for its focus on data integrity, but also skepticism over its use of noisy GitHub-hosted VMs instead of dedicated hardware. Commenters probe how realistic the tests are (e.g., scrubbing after deliberate block corruption, bypassing page cache, custom latency metrics) and suggest additional scenarios, hardware, and clearer data presentation. Much of the debate centers on trade‑offs between performance, reliability, and long‑term support, including concerns over bcachefs’s removal from mainline Linux, Btrfs’s unresolved RAID‑5/6 and robustness issues, and ZFS’s licensing and packaging constraints on Linux distributions.

Benchmark scope & methodology

  • Benchmarks target multi-device, CoW filesystems and RAID-like setups (including integrity / scrub behavior), using fio for throughput, IOPS, and fsync latency.
  • Several workloads are custom (e.g., a 4 KiB write + fsync every 200 ms as a “trivial op” latency probe). These are acknowledged as project-specific, not industry standards.
  • Tests currently run mostly on GitHub-hosted VMs with loop-backed sparse files; the author stresses that only relative “shapes and ratios” should be compared, not absolute MB/s.
  • There is a calibration step to discard the noisiest VMs, and some limited “real hardware” runs exist, with more in progress, including HDD/SSD hybrid and tiered setups.

Data integrity & corruption tests

  • One test overwrites 2 GiB of raw blocks on a single device in a replicated set, then runs filesystem scrubs.
  • Some commenters question whether this is realistically “recoverable” and what “integrity” success/failure actually means.
  • The author clarifies that the label will be renamed to “corruption probe”: “survived” only means a specific file remained readable and unchanged, not that the entire filesystem is fully healthy.

Presentation & usability of results

  • Several people find the page dense and visually hard to scan: small text, too many charts at once, unclear definitions for composite scores like “Overall Core” and “Core I/O”.
  • Others, especially engineers, appreciate the “wall of data” and prefer detail over simplification.
  • Suggestions include clearer descriptions of what’s under test, better explanations for derived metrics, moving run metadata away from the top, and possibly alternative “simplified” views using the JSON data.

Environment realism & hardware coverage

  • Some argue that shared cloud runners with noisy neighbors make results hard to trust, and advocate for dedicated bare-metal testbeds.
  • Others accept the limitations, noting the focus on integrity rather than absolute performance and the difficulty/cost of extensive bare-metal automation.
  • Multiple requests appear for more scenarios: HDD vs SSD, NVMe, different RAID sizes, non-CoW filesystems, dm-integrity, dRAID, F2FS, DRBD, CephFS, and XFS with integrity layers.

Bcachefs, Btrfs, ZFS and alternatives

  • Bcachefs shows strong results in the benchmarks and is praised for features like mixing device tiers and per-file replication, but concerns remain about maturity and recent removal from mainline.
  • Some users are happy with bcachefs via out-of-tree/DKMS on certain distros (e.g., NixOS-based setups, NAS appliances), citing good practical experience.
  • Btrfs is seen as the “default modern” in-tree option but criticized for:
    • Lack of reliable free-space reporting.
    • Fragile behavior when volumes fill up.
    • Weak repair tooling and still-unrecommended RAID5/6.
    • Occasional severe stalls on very large file deletes in some environments.
  • ZFS is widely trusted for data safety but criticized for:
    • Lower performance in some workloads.
    • Licensing friction that keeps it out of many distributions and rescue environments.
    • DKMS lagging behind rapid kernel updates on some distros.
  • Many production-minded commenters still favor simpler stacks like ext4/XFS over LVM/MD RAID for predictability, sometimes augmented with separate compression/dedup layers (e.g., dm-vdo).

Social factors & kernel governance

  • Several comments stress that filesystem “social health” (bus factor, maintainer drama, CoC issues, corporate sponsorship) matters as much as raw performance.
  • Bcachefs’s removal from mainline and prior conflicts with kernel maintainers raise trust and future-stability concerns for some; others argue these are mostly maintainer problems and end users just need good tooling and reliability.
  • There is broad interest in bcachefs eventually returning to mainline once development stabilizes and a broader maintainer team is in place.