Post-mortem for last week's incident at Kagi
A seven-hour outage at paid search engine Kagi, triggered by a single user’s high-volume scraping, has raised questions about how small startups balance lean infrastructure with resilience and abuse protection. Commenters largely praise Kagi’s transparent post-mortem and lean, single-node database setup, but highlight gaps in rate limiting, monitoring, and incident response—especially around status page accuracy and SRE maturity. The incident also fuels debate over what “unlimited” usage really means, and whether services should more clearly communicate fair-use limits and technical safeguards to prevent similar failures.
Scope and nature of the incident
- Outage lasted ~7 hours, triggered by one paying user performing heavy automated scraping, hitting a “pathological” traffic pattern.
- Service was already handling ~400k searches/day; the incident added ~60k over a short window, stressing a single-core primary DB and connection pools.
- Several commenters note this is a classic “unlimited but not really” / “one user can hose the system” scenario that many startups eventually face.
Infrastructure, scaling, and rate limiting
- Many praise the lean setup (cheapest single-core GCP DB, simple Postgres/Redis-style stack) and argue most teams over-engineer with distributed databases too early.
- Others say that if 60k extra requests can bring the system down, infra is too fragile, and per-user rate limiting and/or Cloudflare‑like fronting should have existed already.
- Strong consensus emerges that all public endpoints need QPS limits and burst controls; some share anecdotes where a single stuck key or poorly designed typeahead took down production.
Observability, diagnosis, and status pages
- Discussion highlights how hard it is, especially for a tiny team, to interpret dashboards, differentiate red herrings, and avoid “being gaslit by your own metrics.”
- People debate manual vs automated status pages:
- Some want auto-updated, metric-driven pages; others describe why that often degenerates back into manual overrides and judgment calls.
- Multiple users were frustrated that the status page stayed green while they saw 500s, undermining confidence.
- Several suggest starting from SLIs/SLOs and building alerts and dashboards (e.g., queries per account, lock/IO wait, 500-rate) around known limits.
“Unlimited” vs abuse and ToS
- One side argues that advertising “unlimited searches” but banning heavy automated use is misleading and feels like a bait‑and‑switch; they want explicit “fair use” wording or numeric caps.
- Others counter that “unlimited for human use” obviously doesn’t include scraping or trying to re-index the service, especially when there is a separate paid API.
User sentiment and product quality
- Many commenters are paying users who love the search quality, customizability (e.g., pinning sites), and transparency of the post‑mortem, and say outages are acceptable learning for an early‑stage startup.
- Some say the downtime made them appreciate Google’s near‑perfect reliability and feel uneasy about depending on a small provider.
- A few report account/login issues and take that as a red flag; others say that’s likely an edge case best handled via support.