JuiceFS is a distributed POSIX file system built on top of Redis and S3

Original Article ↗ Hacker News Discussion ↗

ZeroFS vs JuiceFS Performance & Architecture

  • A linked benchmark claims ZeroFS dramatically outperforms JuiceFS on small-file workloads and needs only S3, no external DB.
  • Several commenters are skeptical: the reported scale of failures and speed differences for JuiceFS seem implausible without misconfiguration; the JuiceFS setup isn’t fully documented, so results aren’t easily reproducible.
  • Others report independently that JuiceFS can perform very poorly on torrents / many tiny files, supporting the idea that this class of workload is hard.
  • Key architectural difference: ZeroFS is effectively single-writer (plus read-only mounts) and thus limited by one node’s bandwidth; JuiceFS clients write directly to S3, scaling horizontally as long as the metadata engine keeps up.

Licensing and Adoption

  • JuiceFS is Apache-2.0. ZeroFS is AGPL/commercial, which some see as restrictive or risky for large companies, even if unmodified use is permitted.
  • This is called out as an important axis in comparisons.

Metadata Engines & Durability

  • JuiceFS’s design hinges on an external metadata store; supported options include Redis, SQL DBs, TiKV, FoundationDB, etc. Choice strongly affects latency and scalability.
  • Some distrust Redis for durability and want benchmarks using “serious” stores; others argue Redis with persistence is reliable and very fast.
  • Concern: data blocks in object storage are in a custom format; loss of metadata can mean losing the whole filesystem, similar to other filesystems that separate data and metadata. JuiceFS can back up metadata to S3, but this risk must be managed.

Use Cases and Workloads

  • Reported successful uses: multi-petabyte/HPC-like workloads, Kubernetes RWX volumes, high-bandwidth access backed by MinIO/Valkey, and as a building block in other systems (e.g., custom SQLite+Litestream metadata).
  • Reported limitations: AI training performance issues leading some to abandon it; poor behavior with very many small files; not recommended as a primary database filesystem except for backups/temporary use.

POSIX Semantics, FUSE, and NFS Replacement

  • JuiceFS advertises strong close-to-open consistency and atomic metadata operations; some compare this favorably to other cloud filesystems, but doubt it’s safe for “real” databases.
  • FUSE is described as full of gotchas and compatibility work; achieving good performance and full POSIX behavior is nontrivial.
  • Several people want an NFS replacement, but others caution that FUSE-based systems have meta-store bottlenecks and surprising performance cliffs.

Operational Complexity, Scale & Alternatives

  • Some prefer mature systems like CephFS, Lustre, or EFS despite cost/complexity; others see JuiceFS as a much easier drop-in that can scale to hundreds of PB with an appropriate metadata backend.
  • A few argue that POSIX-on-S3 is fundamentally a workaround, and that large-scale systems should speak S3 natively with separate metadata in a DB.
  • Alternative approaches mentioned include ZeroFS, Archil (SSD-backed distributed store with S3 integration), and systems like Object Mount that avoid separate metadata by keeping a 1:1 file–object mapping.