Saving cloud costs by writing our own database
A startup claims to have cut its AWS Aurora bill by roughly 98% by replacing it with a custom append-only storage engine on EBS for geospatial time-series data. Commenters question the trade-offs around durability, availability, and long‑term maintenance, noting that the system is closer to a specialized log format than a full ACID database and that off‑the‑shelf tools like ClickHouse, Kafka, or TimescaleDB might offer similar benefits with fewer risks. The exchange highlights a broader tension between building narrowly tailored infrastructure to save cloud costs and relying on mature, general‑purpose database systems that are easier to hire for and reason about.
Headline and Savings Claims
- Several commenters question “5000% savings,” noting it should be ~98% and joking about saving “more than 100%.”
- Some see the comparison as unfair: Aurora provides HA and rich features, while the custom solution is far narrower.
What They Actually Built (Database vs Log File)
- Many argue it’s not a general-purpose database but an append-only binary log with minimal query features.
- Others respond that “database” is broad; any structured storage plus retrieval qualifies, and lots of real systems are append-only logs plus indexes.
- The article is seen as mixing talk of ACID and geospatial features with a final design that omits most of that, which some find misleading.
Alternatives Suggested
- Repeated suggestions: ClickHouse, TimescaleDB, Cassandra/Scylla, Elasticsearch, Kafka/Pulsar, Parquet-on-S3 with Trino/Athena, DuckDB, TileDB.
- Several argue ClickHouse in particular is a strong fit for high-volume time-series with geo support and cold storage on S3.
- Others note that simply running Postgres/SQLite on EC2 or using existing time-series engines would have avoided custom storage.
Reliability, Durability, and ACID Concerns
- Concerns about EBS latency jitter, fsync failures, snapshot consistency, and lack of documented checksums or corruption handling.
- Critiques that non-ACID append-only formats must still handle partial writes and silent corruption; “battle-tested” DBs solve these edge cases.
- Aurora’s multi-AZ replicated storage is contrasted with a single-AZ EBS volume; cross-AZ availability strategy is unclear.
- Tension noted between “replay exact seconds before an accident” and accepting up to a second of buffered data loss.
Performance, Scaling, and “Unlimited Parallelism”
- Write throughput (tens of KB–MB/s) is considered modest; many off-the-shelf systems could handle it if batched.
- Some question how “unlimited parallelism” works when EBS volumes are AZ-local and single-attached by default; multi-node story and sharding are unclear.
- Read-path scaling is questioned: scanning huge log files for queries may become a bottleneck without indexes.
Cost vs Engineering Effort
- One camp: $10k/month Aurora vs $200/month EBS justifies a bespoke engine, especially if scaled 10–50× over time.
- Opposing camp: initial build, ongoing maintenance, bus factor, and PM/coordination costs may outweigh cloud savings, especially if a capable engineer is diverted from core product work.
- Some argue Aurora and other managed DBs also require expertise, so engineering cost isn’t zero there either.
Philosophical Split: Build vs Buy
- Enthusiasts praise focused engineering: only implement needed features, avoid overpaying for general-purpose platforms, and understand your own system.
- Skeptics worry about “Not Invented Here,” underestimated complexity, and future feature creep gradually forcing them to reinvent a real database.
- Broader criticism of cloud lock-in and “just stitch together managed services” culture vs rebuilding core infrastructure to regain cost control and flexibility.