Why we use our own hardware

Original Article ↗ Hacker News Discussion ↗

Own Hardware vs Cloud: Big Picture

  • Many agree Fastmail’s choice fits its workload: email has predictable, steady load and is very storage‑heavy, so owning hardware and using colo is economical and performant.
  • Others stress that for many orgs, especially without deep infra talent, public cloud’s trade‑off (higher unit cost, lower operational burden) is still attractive.

Economics, CapEx/OpEx, and Middle Grounds

  • Repeated claim: cloud can be ~3× more expensive for steady workloads, especially once you factor in IO, bandwidth, and egress.
  • Counterpoint: when you include salaries, 24/7 ops, procurement delays, and compliance work, cloud TCO can look good, especially for larger regulated orgs.
  • CapEx vs OpEx and internal politics are major drivers: cloud spend is easier to approve, hides infra cost behind APIs, and avoids hardware purchase cycles.
  • Several people highlight “middle ground” options: colocation, rented dedicated servers, or hybrid (base load on metal, bursts in cloud).

Reliability, Redundancy, and Security

  • Fastmail staff in the thread confirm multi‑datacenter replication with async failover; redundancy is at the application/protocol level rather than via Ceph‑style distributed storage.
  • Some argue big clouds are more secure and robust than most corporate DCs (formal methods, strong KMS, physical/logical separation).
  • Others push back: hypervisor bugs, centralization, and government access are concerns; app‑level security remains the real weak point in either model.

Storage Stack: ZFS, SSDs, and Email Layout

  • Fastmail uses ZFS on NVMe SSDs broadly (mail, DB, logs, backups), reporting far lower failure rates than with HDDs and plenty of write endurance.
  • They rely on Cyrus IMAP’s own replication for mail, using ZFS replication only for logs, partly due to known ZFS encryption + send/receive bugs.
  • Discussion dives into ZFS behavior under high fill ratios, unlink performance, parallel deletion tricks, and caution about ZFS native encryption in production.

Cloud Skills, Sysadmins, and Tooling

  • Several predict a quiet resurgence of classic sysadmin skills as cloud costs bite and “cloud‑only” knowledge doesn’t transfer well.
  • Others say modern DevOps + IaC on cloud enables one team to manage far more infra, bypassing slow internal IT and raising delivery speed.
  • There’s broad agreement that both cloud and on‑prem are complex; each just concentrates difficulty in different places (hardware vs IAM/VPC/RDS/etc.).

Fastmail Service Feedback

  • Many commenters are enthusiastic long‑time users: praise performance, UI, mobile apps, masked emails, domain handling, transparency, and support.
  • Critiques include weaker spam filtering than Gmail/iCloud for some, slow or limited search, and occasional app bugs; one user unhappy with a Pobox migration.