SeaweedFS fast distributed storage system for blobs, objects, files and datalake
SeaweedFS, an open-source distributed storage system inspired by Facebook’s Haystack, is attracting attention as a fast, cost-effective alternative to S3 and other self-hosted object stores for workloads involving billions of small or medium-sized files. Commenters report strong performance and reliability at scale, especially versus MinIO in some HDD-heavy setups, but note trade-offs around setup complexity, sparse operational documentation, incomplete POSIX semantics, and the usual challenges of running distributed storage safely in production. The thread situates SeaweedFS among alternatives like Ceph, Garage, JuiceFS, and traditional ZFS, highlighting how API compatibility, metadata design, erasure coding, and operational burden shape the choice of storage backend.
Real-world deployments and performance
- Multiple users report SeaweedFS running in production or labs: Kubernetes PVs, home labs, 250TB+ of audio, 50TB+ of game replays, billions of thumbnails, and multi‑billion small files.
- Consistently praised for performance with many small/medium objects (thumbnails, XML, PDFs), good latency even at high percentiles, and efficient HDD usage.
- Some note it “just works” once configured, with years of stable operation even at smaller scales (~250k objects).
Comparison with alternatives
- Frequently compared to MinIO: historically MinIO was weaker on tiny files, though newer versions improved; one team still found SeaweedFS faster for >100TB HDD workloads, especially given MinIO’s erasure coding overhead and rigid expansion model.
- Garage is suggested as a simpler S3-only option, with easier-to-read code but no erasure coding and AGPL licensing.
- Ceph is viewed as powerful but heavy/complex; one comment claims SeaweedFS has much lower metadata overhead and faster writes with its volume-level erasure coding.
- Longhorn is mentioned as block storage (EBS-like) rather than S3-like.
- JuiceFS requires a separate backing store (e.g., S3, SeaweedFS) and is not a standalone SDS; one tester saw correctness issues with slow backends.
Architecture and design
- Built on a Haystack-style append-only blob store: large “volumes” with packed blobs and separate metadata, targeting O(1) disk/network ops per access.
- Higher-level file and S3 layers are metadata services on top of blobs; metadata can be stored in different backends (Postgres, Cassandra, Redis, etc.).
- Volumes are append-only with a “vacuum” process to reclaim space after deletions.
Operational challenges and reliability
- Setup, tooling, and CSI drivers are described as obscure or clunky; CSI sidecars can be resource-heavy.
- Some past issues with under-replicated objects during heavy concurrent writes; believed fixed in newer releases.
- Running databases like Postgres over the SeaweedFS CSI mount failed for one user; others warn that databases on generic network filesystems are risky.
When to use vs cloud S3
- For workloads fully on AWS, commenters see little advantage over S3.
- For on-prem or non-cloud deployments, SeaweedFS is attractive for avoiding egress costs and leveraging cheap HDDs.
Documentation and clarity gaps
- Repeated calls for better docs on: small-file behavior, fragmentation and vacuum impact, scrubbing/bitrot repair, upgrade procedures, and detailed trade-offs between filer backends.
- Some architectural explanations around “blobs vs files vs objects” are viewed as unclear or incomplete.