Why we use our own hardware
An email provider’s decision to run its own hardware instead of relying on AWS‑style cloud platforms has reignited debate over when cloud actually makes technical and economic sense. Commenters highlight how colocation, SSDs, ZFS, and protocol‑level replication can deliver high performance, reliability and lower long‑term costs for predictable workloads like email, while cloud excels at rapid provisioning, global reach, managed services and sidestepping internal bureaucracy. Many see a growing split: large or heavily regulated organizations and fast‑moving startups may benefit from cloud and PaaS, whereas mature, infrastructure‑savvy teams can often operate their own servers more efficiently and with greater control.
Own Hardware vs Cloud: Big Picture
- Many agree Fastmail’s choice fits its workload: email has predictable, steady load and is very storage‑heavy, so owning hardware and using colo is economical and performant.
- Others stress that for many orgs, especially without deep infra talent, public cloud’s trade‑off (higher unit cost, lower operational burden) is still attractive.
Economics, CapEx/OpEx, and Middle Grounds
- Repeated claim: cloud can be ~3× more expensive for steady workloads, especially once you factor in IO, bandwidth, and egress.
- Counterpoint: when you include salaries, 24/7 ops, procurement delays, and compliance work, cloud TCO can look good, especially for larger regulated orgs.
- CapEx vs OpEx and internal politics are major drivers: cloud spend is easier to approve, hides infra cost behind APIs, and avoids hardware purchase cycles.
- Several people highlight “middle ground” options: colocation, rented dedicated servers, or hybrid (base load on metal, bursts in cloud).
Reliability, Redundancy, and Security
- Fastmail staff in the thread confirm multi‑datacenter replication with async failover; redundancy is at the application/protocol level rather than via Ceph‑style distributed storage.
- Some argue big clouds are more secure and robust than most corporate DCs (formal methods, strong KMS, physical/logical separation).
- Others push back: hypervisor bugs, centralization, and government access are concerns; app‑level security remains the real weak point in either model.
Storage Stack: ZFS, SSDs, and Email Layout
- Fastmail uses ZFS on NVMe SSDs broadly (mail, DB, logs, backups), reporting far lower failure rates than with HDDs and plenty of write endurance.
- They rely on Cyrus IMAP’s own replication for mail, using ZFS replication only for logs, partly due to known ZFS encryption + send/receive bugs.
- Discussion dives into ZFS behavior under high fill ratios, unlink performance, parallel deletion tricks, and caution about ZFS native encryption in production.
Cloud Skills, Sysadmins, and Tooling
- Several predict a quiet resurgence of classic sysadmin skills as cloud costs bite and “cloud‑only” knowledge doesn’t transfer well.
- Others say modern DevOps + IaC on cloud enables one team to manage far more infra, bypassing slow internal IT and raising delivery speed.
- There’s broad agreement that both cloud and on‑prem are complex; each just concentrates difficulty in different places (hardware vs IAM/VPC/RDS/etc.).
Fastmail Service Feedback
- Many commenters are enthusiastic long‑time users: praise performance, UI, mobile apps, masked emails, domain handling, transparency, and support.
- Critiques include weaker spam filtering than Gmail/iCloud for some, slow or limited search, and occasional app bugs; one user unhappy with a Pobox migration.