Working with Files Is Hard (2019)

Original Article ↗ Hacker News Discussion ↗

File systems and their APIs turn out to be surprisingly hard to use correctly, even for database and storage experts, because POSIX semantics, hardware caches, and power-loss behavior make durability and atomicity guarantees tricky or misleading. Commenters examine research showing that nearly all major databases and filesystems misuse these APIs in subtle ways, debate alternatives such as file-level atomic operations or barrier syscalls, and compare behaviors across Linux, BSD, Windows, NFS, and ZFS. A recurring theme is the gap between what applications assume storage will do, what hardware and kernels actually guarantee, and how much complexity is acceptable outside of high‑reliability systems.

POSIX filesystem APIs and why they’re “hard”

  • Research referenced in the thread shows many prominent systems (DBs, VCSs, distributed systems) misuse file APIs, even with expert developers.
  • Many argue the core problem is the POSIX model: old, entrenched, and underspecified on key semantics (ordering, atomicity, error propagation).
  • Others counter that APIs can’t be “impossible to misuse” and that many apps reasonably assume simpler conditions (e.g., single-writer).
  • Some see this as a “Worse is Better” outcome: cheap-to-implement semantics outcompeted safer designs.

Alternative abstractions and atomicity models

  • Several proposals: whole-file atomic writes via copy-on-write, atomic appends, treating files as atomic block maps, or transactional/DB-like semantics at the filesystem level.
  • Advocates claim this would remove large bug classes and simplify reasoning about shared files.
  • Critics raise concerns: multi‑GB files, extra space for copy-on-write, SSD wear, multi-process access, and difficulty retrofitting existing software and filesystems.
  • There’s discussion of database-style transactions (and deadlocks), with suggestions that MVCC-like approaches could mitigate some issues.

Barriers, fsync, and storage hardware behavior

  • Debate over why Linux still lacks a non-flushing barrier syscall to separate ordering from durability; some think it would significantly help databases.
  • Others note a prototype exists in research code but hasn’t been adopted, possibly due to limited benefit, SSD-era tradeoffs, or maintenance burden.
  • NVMe, FUA, and controller caches complicate “flush” guarantees; buggy hardware and lack of proper FUA support are cited.
  • It’s emphasized that some devices can lose or corrupt data even after flush, and that sector-atomic assumptions are not universally valid (e.g., certain non-volatile memories, commodity flash).

Windows, C libraries, and API evolution

  • Windows file APIs are described as somewhat safer/clearer but slower, with features like IOCP and strict locking on executables.
  • Lack of open research on Windows filesystems is attributed to NDAs and corporate control over publication.
  • An analogy is drawn to unsafe C standard functions: attempts to “stage in” safer alternatives are messy, non-portable, and often misunderstood.

Databases, SQLite, and failure handling

  • SQLite is praised as a safer choice when persisting local state, especially in specific modes (e.g., WAL, strict synchronous settings).
  • Later research simulating fsync errors found that major systems (Redis, SQLite, LevelDB, LMDB, PostgreSQL) still mishandle some failure modes.
  • Some systems deliberately rely on de facto hardware guarantees (sector-atomic writes), which may fail on certain devices.

NFS and distributed semantics

  • NFS is criticized for breaking important file guarantees (append, exclusive create, sync flags, locks, inotify), especially across UID mappings.
  • This leads to surprising behaviors such as read access changing after a successful open, complicating userland code.

Filesystem-specific behaviors and reliability

  • ext4 has special logic to make common “rename for atomic replace” patterns safer.
  • ZFS is discussed as robust but with Linux-specific issues under heavy load that may involve IO schedulers and external factors; there’s ongoing debugging.
  • Some report more corruption with modern filesystems than FAT; others stress that power loss and hardware flaws are fundamental and must be engineered around, not merely “fixed” operationally.

Meta observations

  • Many note that filesystems and storage “mostly work” until rare, harsh failure conditions.
  • There’s tension between accepting imperfect semantics for 95% of use cases and demanding stronger guarantees for critical systems.