Ceph: A Journey to 1 TiB/s
Engineers dissect a Ceph storage cluster that achieves over 1 TiB/s read throughput using 68 NVMe-heavy servers, focusing on where the true bottlenecks lie (network vs. PCIe vs. CPU) and how close the setup comes to theoretical limits. Many weigh Ceph’s strengths—resilience, scale-out growth, and feature set—against its complexity and latency, especially for home labs and database workloads, and compare it with alternatives like MinIO, SeaweedFS, Longhorn, GlusterFS, ZFS, btrfs, and EOS. The conversation also touches on practical hardware choices, from 100 GbE+ switching and NICs down to Raspberry Pi clusters, and on Ceph’s history and real-world use in environments like OpenStack, Kubernetes, CERN, and custom AWS deployments.
Hardware & networking for 1 TiB/s
- Discussion notes that 800 Gbps switches exist, but 800 G NICs are not yet practically available; PCIe lane counts and generations are a constraint but not a hard blocker if 32 lanes are used.
- Most contributors assume modern 100 GbE switches (pizza-box / leaf-spine / Clos fabrics) can handle full line rate on all ports; any switch that cannot is considered “toy” gear.
- Used 40/56/100 GbE switches are now relatively cheap; some “bargain” MikroTik and used data-center switches are mentioned as homelab-friendly.
- Analysis of the benchmark cluster concludes the network, not the NVMe drives, is the bottleneck; measured 1 TiB/s is ~65% of theoretical network capacity across nodes.
Ceph performance characteristics
- Ceph offers strong scalability and resilience, but with notable CPU overhead and relatively weak latency, especially versus local NVMe.
- Benchmark details highlight compiler optimization flags and OSD threading/IOMMU contention as surprisingly large performance factors; developers acknowledge the threading model needs redesign.
- For transactional databases, several participants warn Ceph’s latency is often too high, though some report acceptable block (RBD) latency with good hardware.
Ceph in homelabs
- Many caution against Ceph at home unless the goal is learning; hardware minimums (multiple nodes, fast networking, non-SMR disks) and complexity are real.
- Others successfully run Ceph on Proxmox clusters, mini‑PCs, NUCs, and even Raspberry Pis/ODROID boards, accepting lower performance and higher latency.
- A recurring theme: Ceph is excellent for resilience and elasticity, not raw speed or simplicity.
Alternatives & related systems
- Common homelab alternatives: SeaweedFS, Longhorn, MinIO, Garage, GlusterFS (though Red Hat support is ending), EOS, ZFS (especially mirror vdevs), Btrfs, LVM+mdadm, and SnapRAID.
- Trade-offs discussed:
- MinIO: good S3 object store but touchy about cluster size changes and demands fast disks.
- Garage: simple, duplication-only; fine for homelabs, not large datasets.
- ZFS mirrors: easy incremental expansion and fast resilvering but 50% space efficiency.
- EOS at CERN: tuned for physics workloads, complements Ceph rather than replaces it.
Operational complexity & rationale
- Ceph is viewed as “don’t deploy unless you mean it”: complex but uniquely good for large, cheap, linearly scalable, highly available storage on commodity hardware.
- It underpins OpenStack/Kubernetes deployments, in-house EBS‑like systems, and high‑IOPS caches, where users report big cost and performance gains over cloud block storage.