Incident Report: Railway Blocked by Google Cloud [resolved]

Original Article ↗ Hacker News Discussion ↗

Incident and suspected cause

  • Railway’s outage traced to its GCP account being put into a “restricted” state; a GCP project was reportedly deleted without warning, removing CloudSQL, overflow VMs, and API access.
  • Railway reps say they had prior assurances from Google after an earlier auto‑rate‑limit incident that this wouldn’t recur, and that restoration took minutes once a bug was filed, but damage to customers lasted hours.
  • Exact trigger is still unclear in the thread (possibilities floated: abuse reports, payment issues, anti‑fraud/AI systems, or customer workloads), and several commenters stress we only see one side.

Railway architecture and “not a cloud on a cloud”

  • Railway markets itself as owning its own metal and not “building a cloud on another cloud.”
  • Commenters discover core databases and some networking still depended on GCP, contradicting that narrative in their view.
  • Railway explains they exited most compute to their own DCs (plus AWS), but left DBs on CloudSQL for HA/replication and to avoid circular dependency on their own infra; in hindsight this became the critical single point of failure.
  • Some see this as understandable technical tradeoff; others call it deceptive or dangerously backwards (DB last to migrate).

Trust in GCP: bans and support

  • Many recount prior GCP suspensions (including smaller accounts and a Korean government org) and the UniSuper incident where a misconfig deleted a whole private cloud subscription.
  • General themes: aggressive automated enforcement, weak human support/CSM effectiveness, and fear that even high‑spend accounts can be “auto‑yeeted.”
  • A minority report good GCP relationships and argue such blow‑ups usually follow earlier warning signs or poor account hygiene.

Redundancy, multi‑cloud, and backups

  • Strong chorus: “all eggs in one basket” is risky, especially for critical control‑plane components like auth, DNS, and primary DBs.
  • Others counter that true multi‑cloud is extremely rare, complex, and often unjustified for startups; you typically “start with one egg.”
  • Several emphasize off‑provider backups and separate billing entities; 3‑2‑1 backup interpreted as “different accounts/providers,” not just extra buckets.
  • Discussion notes that shutting an account/subscription can be a global single point of failure despite multi‑region setups.

Comparisons to AWS/Azure and other hosts

  • Many say they’ve never seen AWS/Azure silently nuke accounts at this scale; AWS is criticized for regional outages (especially us‑east‑1) but praised for warnings and softer enforcement.
  • Some note other providers (Hetzner, OVH) are also aggressive on KYC/abuse; AWS/Azure are framed as the safer outliers for account risk.
  • Alternatives floated: Render, Vercel, Fly.io, DigitalOcean, Hetzner, Coolify, self‑hosted/colo, even rsync‑style offsite storage.

User impact and reactions

  • Hobbyists and small customers experienced long downtime, invalid TLS certs, and 502s; some had to manually redeploy even after Railway marked the incident “resolved.”
  • New and existing customers describe this as a “wake‑up call”; several immediately migrated to other platforms, saying trust is broken.
  • Others express continued sympathy for Railway but resolve not to run serious businesses on such a young platform.

Abuse, anti‑fraud, and free tiers

  • Some operators complain about heavy spam/abuse from Railway IPs and say its abuse prevention is weak.
  • Railway previously acknowledged internal anti‑fraud misfires that “hard killed” legitimate workloads.
  • Broader debate: free/cheap compute inevitably attracts abuse; strict KYC and anti‑fraud reduce that but hurt growth and UX.