S3 is files, but not a filesystem

Original Article ↗ Hacker News Discussion ↗

What S3 Is (and Isn’t)

  • Broad agreement: S3 is an object store / key–value store, not a traditional filesystem.
  • Core model: flat mapping of keys → objects; keys are opaque strings, operations are whole-object GET/PUT/DELETE.
  • POSIX-style semantics (inodes, directories as first‑class objects, partial writes, atomic renames) are absent by design.
  • Some argue S3 still counts as “a kind of filesystem” in the broad sense (“system that manages files”), others insist that dilutes the term and misleads users into wrong assumptions.

“Folders” and Path Semantics

  • S3 has no real directories. “Folders” in the console are UI sugar over key prefixes, sometimes implemented with 0‑byte “marker” objects.
  • Consequences:
    • You can have keys like dir, dir/file, dir//file simultaneously; slashes are not special to S3.
    • Empty “folders” vanish when the last object with that prefix is deleted.
    • Renaming a “folder” requires copying and deleting every object with that prefix (O(number of objects) and non‑atomic).

Operations, Performance, and Listing

  • No in‑place modification or append; you must re‑upload altered content or chunk at a higher layer.
  • Copy and rename are implemented as copy+delete, with cost and latency scaling with object size.
  • Listing is a major semantic and performance difference:
    • S3 lists by prefix in lexicographic order and supports pagination; can be very efficient for prefix‑based scans and partitioning.
    • Recursive, POSIX‑style directory walks are slow and require many sequential list calls; deleting or emptying large buckets can be expensive and complex.

Databases and Filesystem Layers on S3

  • Running traditional databases directly on S3 is widely viewed as a bad fit: latency, lack of partial writes, no atomic “write if not exists”.
  • Successful patterns:
    • Systems that are append‑only or treat files as immutable parts (e.g., analytical engines, table formats like Delta/Iceberg/Hudi).
    • Architectures that use S3 as a data plane with a separate, strongly consistent metadata/index layer (e.g., DBs using WAL on S3 plus local caches, or external metadata services).
    • FUSE and VFS layers (e.g., rclone, custom filesystems) that emulate POSIX by caching locally, chunking, and managing metadata elsewhere—useful but complex and leaky.

Durability, Cost, and Alternatives

  • S3’s durability is strongly praised; described as industry‑leading with heavy use of checksums and replication, though third‑party, Jepsen‑style validation is not cited.
  • S3 is generally seen as cheap for large, cold datasets; others note cheaper object stores exist but may trade off reliability or features.
  • For true filesystem semantics and low latency, commenters point to EBS/EFS/FSx or on‑prem filesystems; for “S3‑like but simpler” local setups, object‑storage servers or custom layers are suggested.