Binance built a 100PB log service with Quickwit
Binance’s move to store and search 100 PB of application logs using the Quickwit search engine prompts broader questions about how much logging is actually necessary and how to do it cost‑effectively. Commenters dissect the technical approach—object storage, compression, indexing, and query performance at multi‑petabyte scale—while contrasting it with alternatives like BigQuery, ClickHouse, Kafka pipelines, or custom storage. A major theme is the trade-off between rich, long‑term logs for observability, auditing, and forensics versus relying more heavily on metrics, sampling, and stricter log hygiene to avoid runaway complexity and cloud bills.
Scale and Use Case for 100 PB of Logs
- Logs are primarily application/observability logs (what a smaller org might send to Datadog), not blockchain or transaction ledgers.
- Binance reportedly logs huge volumes from APIs, microservices, HFT bots, and user activity; some estimate trillions of events and millions of orders per second at peak.
- Use cases include debugging, operational observability, customer support (“what happened to this user 2 months ago?”), and regulatory/audit investigations that may arrive months late.
- Several commenters are skeptical that sub‑second search and months of full‑fidelity logs are really necessary; others argue finance and security/liability make deep retention and fast search justifiable.
Quickwit vs. Elasticsearch and Other Tech Choices
- Elasticsearch was seen as too expensive and complex at this scale; Quickwit stores indices directly on object storage (e.g., S3) and queries them in place, with optional RAM caching.
- Quickwit uses Zstd compression, Lucene-like indexing, and object-storage-friendly dictionaries; building inverted indices is described as strongly CPU‑bound.
- Reported ingest: ~1.6 PB/day into ~20 PB compressed (≈5:1); some consider this compression weak for logs and point to more aggressive log-specific schemes.
- Regex search is limited due to the chosen dictionary structure; prefix and tokenized search are supported.
Cost, Storage, and Infrastructure Debates
- Rough cost estimates: ~US$460k/month for 20 PB S3, ~US$100k/month for compute; discounts, spot, and alternative storage classes can reduce this.
- Others argue self‑hosted HDD arrays with erasure coding and colo could be significantly cheaper over 5 years, but with higher operational complexity and IOPS constraints for “hot” data.
- There is extensive discussion of write amplification from verbose JSON logs; many argue changing logging formats and sampling could save more than sophisticated indexing.
Logs vs. Metrics vs. Traces
- Large sub‑thread argues logs should not be the primary source of operational truth: metrics are cheaper, better for SLAs/alerts, and easier to keep organized.
- Opposing view: for application debugging, forensics, and financial traceability, detailed logs are irreplaceable; metrics and even traces can be derived from them.
- Common suggested compromise: structured logs, strict retention windows, aggressive sampling/tail-sampling (especially for “OK” traces), and separate high‑assurance audit trails for money flows.