Billion file filesystem

Original Article ↗ Hacker News Discussion ↗

Pushing a Linux filesystem to hold a billion empty files becomes a stress test of metadata overhead, directory indexing, and tooling limits rather than raw storage capacity. Commenters compare ext4, XFS, btrfs, ReiserFS, ZFS, and NTFS under heavy small-file workloads, noting problems like inode bloat, slow directory operations, and pathological deletion times. Many argue that databases or formats like SQLite, specialized filesystems, or alternative layouts (e.g., sharded directories or FUSE/virtual filesystems) are often better suited than general-purpose filesystems for workloads involving millions or billions of tiny objects.

Rust vs shell/Python for this task

  • Several comments note the Rust program is conceptually simple (a few execs and a loop) and could be reproduced in a short shell script.
  • Others argue Rust’s benefits show up in performance, parallelism, and maintainability once datasets are large (e.g., millions of URLs), even if a quick prototype in bash/Python is shorter.
  • Some suggest splitting “mkfs/mount” from the Rust logic to align with “do one thing well.”

Filesystem behavior with huge directories

  • Creating a billion files in one directory is seen as a distinct stress case; directory lookup and listing become key bottlenecks.
  • Benchmarks are shared for ls on 1M–10M files; unsorted listing (ls -U) and single-column output (-1) are dramatically faster than default ls.
  • Ext4’s htree indexing reduces kernel-side scaling problems, but userspace tools that slurp entire directories into memory still misbehave.

Space and metadata overhead

  • Ext4’s per-file overhead (~296 bytes for an empty file here) is linked mainly to inode size (default 256 bytes) plus directory and other metadata.
  • Smaller inodes (e.g., 128 bytes) can halve this but reintroduce 32‑bit timestamp/Y2038 limits.
  • Some specialized systems (e.g., SeaweedFS) claim far lower per-file metadata.

Real-world experiences with many small files

  • Multiple anecdotes: simulations producing 100k+ small files, NTFS directories with millions of files becoming painfully slow to delete or copy, and Linux systems behaving oddly with tens of thousands of files per directory.
  • One case describes ext4 directory hash structures growing so large over billions of file churns that ENOSPC appears despite ample free space, motivating a switch to XFS.

Best storage approach: FS vs DB vs specialized FS

  • Consensus: general-purpose filesystems are not “great” for huge numbers of tiny files.
  • Databases (SQLite/Postgres) often outperform filesystems for many small objects; moving one DB file is far easier than millions of files.
  • Object stores or read-only formats (SquashFS) are suggested when appropriate.
  • ReiserFS historically excelled at tiny files; Btrfs and others have optimizations, but trade-offs remain.

Alternative approaches and tips

  • Suggestions include a FUSE filesystem that virtually exposes a billion files without actually creating them.
  • Various command-line patterns (ls -U1 | wc -l, find ... -printf) and references to getdents() are shared for handling large directories more efficiently.
  • Some discussion touches on ZFS, Hammer2, and the broader “filesystem vs database” semantic and performance trade-offs.