AWS: Inaccurate Estimated Billing Data – $1.7 billion
A widespread AWS billing glitch briefly showed customers estimated charges in the billions and even trillions of dollars, triggering panic among hobbyists and enterprises who normally pay only cents or a few hundred dollars a month. Commenters point to a likely unit-pricing bug in S3 or the billing subsystem (e.g., confusing bytes with gigabytes), and criticize AWS for weak safeguards, lack of hard spending caps, and poor communication while estimates were wrong. Many see the incident as a warning about over-reliance on opaque, AI- and automation-heavy cloud billing systems where subtle errors could be much harder to detect.
Incident overview
- AWS Billing Console showed wildly incorrect estimated charges: from tens of thousands up to billions, trillions, and even quadrillions of dollars.
- Many cases were tied to S3; AWS Health reported “Inaccurate Estimated Billing Data” due to a unit-pricing issue in the estimated billing computation subsystem and paused estimate updates while recomputing.
- Multiple users confirmed their actual usage/costs were normal; this affected estimates and budget alerts, not real invoices (yet).
User impact and reactions
- Hobby and low-usage accounts (often <$1/month) saw estimates in the millions–trillions, triggering panic, trembling, cold sweats, and near–panic attacks; a few immediately deleted infra or closed dormant accounts.
- Some found it funny once they saw obviously impossible numbers (e.g., above AWS’s annual revenue), but others noted smaller, “plausible” errors (5–6 figures) are actually more frightening.
- Several worried about real health consequences and drew parallels to past financial app UI errors that preceded suicides.
Suspected technical cause
- Multiple commenters converged on a unit-conversion bug: bytes being billed at per‑GB rates (off by ~2^30), or a similar metering–pricing mismatch.
- An ex-AWS engineer described a prior real incident: a cents-per-GB price accidentally applied per byte, creating multi‑million‑dollar bills in hours.
- Discussion emphasized brittle separation between metering and pricing plans, weak typing for units, and lack of robust end‑to‑end tests and anomaly detection (“pause billing if bills jump by orders of magnitude”).
Quality, AI, and “vibe coding”
- Strong criticism that something so basic in a utility-like service passed tests, reviews, and monitoring.
- Many blamed or suspected overuse of generative AI/“agentic” systems in billing, citing AWS job ads marketing AI-driven, autonomous billing and validation.
- Others pointed out humans have long shipped similar bugs; AI mainly accelerates volume and reduces careful review.
Cloud billing risk and safeguards
- Repeated frustration that AWS offers alerts but no true hard spending caps or real‑time cutoffs; people fear compromised keys or misconfigurations far more after this.
- Proposed mitigations: spend-limited cards or disabling autopay; stronger anomaly detection; options to shut down compute or freeze storage on threshold breaches.
- Some argue AWS’s behavior justifies treating large clouds like regulated utilities; others say competition means customers should move if they’re unhappy.
Trust, communication, and lessons
- Many were angrier about communication than the bug itself: alert emails went out with no immediate clarifying follow‑up; the incident banner was buried in the health/support pages, not the billing dashboard.
- The episode deepened skepticism about whether past bills were ever fully correct and reinforced calls for detailed usage reports, independent audits, and better internal QA.