Sharing details on a recent incident impacting one of our customers

Original Article ↗ Hacker News Discussion ↗

Google Cloud’s postmortem on accidentally deleting a large financial customer’s managed VMware environment has raised serious questions about its operational rigor and data-deletion safeguards. Commenters criticize the reliance on an internal tool whose default behavior scheduled auto-deletion without clear human review or customer notification, and view Google’s remediation as narrowly focused on this one failure mode rather than systemic protections. The incident is used to argue for cross-cloud or off-cloud backups, stronger “soft delete” and approval processes for destructive actions, and to compare Google Cloud’s perceived maturity and trustworthiness unfavorably with AWS and, to a lesser extent, Azure.

Incident nature and scope

  • Many were surprised the failure was entirely on Google Cloud’s side, not customer misconfiguration.
  • Clarification that a GCVE (managed VMware) private cloud in one region was deleted, not the entire GCP account, reassured some but still seen as catastrophic.
  • Some doubt Google’s claim that this was the first such incident, arguing it’s unlikely a multi‑billion fund was the first affected.

Root cause, defaults, and internal tools

  • Strong criticism that an internal tool allowed a blank parameter to default to “auto‑delete after a year,” and that this could bypass usual safety checks.
  • Several see this as a systemic process failure, not an isolated bug: internal tools are perceived as under‑scrutinized “tech debt” compared with public APIs.
  • Others note Google’s fix (removing the manual operation and further automating deployment) addresses this specific path but not necessarily broader classes of errors.

Safeguards, deletion semantics, and proposed mitigations

  • Many are alarmed that there was no soft delete, advance notification, or human approval before deleting an active, large tenant.
  • Suggested mitigations:
    • Soft‑delete/“disabled but recoverable” state before permanent deletion.
    • Mandatory pre‑deletion customer notifications, regardless of trigger.
    • Human review for large or in‑use service terminations.
    • Cross‑cloud or external backup options as first‑class features.
  • Debate over soft delete: some see it as “enterprise 101”; others point to complexity and data‑protection concerns, but most agree high‑level soft delete for entire services is reasonable.

Backups and cross‑cloud resilience

  • The customer’s off‑platform or third‑party backups are viewed as the real savior; this prompts strong advocacy for off‑cloud or cross‑cloud backups.
  • Some note regulators force such redundancy in financial sectors; many less‑regulated users likely lack this protection.

Trust in GCP vs AWS/Azure

  • Repeated claims that AWS is still the reliability “gold standard,” Azure has security issues, and GCP lags in maturity and rigor.
  • Some argue all large providers have serious “war stories”; what matters is quality of postmortems and systemic fixes.

Google’s postmortem, culture, and communication

  • Many see the postmortem as defensive, focused on denying systemic issues rather than deep remediation.
  • Critiques include:
    • Lack of broader audits of deletion workflows beyond GCVE.
    • Overly self‑congratulatory tone (“most resilient cloud”) despite the failure.
    • Nameless, corporate voice instead of accountable leadership sign‑off.
  • Speculation that the customer received substantial credits or compensation; details remain unclear.