Moving our warehouse off DuckDB
Frequent database and warehouse migrations at analytics company PostHog are raising questions about planning, scalability testing, and the influence of LLM-powered “easy rewrites” on engineering choices. Commenters debate the merits of DuckDB, ClickHouse, Trino, and data lake/table formats for multi-tenant, large-scale workloads, while criticizing the lack of concrete performance data and clear technical justification in PostHog’s blog post about moving off DuckDB. Many see the rapid stack churn as symptomatic of resume-driven development and weak architectural discipline rather than a carefully measured evolution.
Scope and architecture clarifications
- The change concerns a warehouse product offered to customers, not the main analytics database.
- ClickHouse still powers core analytics; DuckDB was used for a separate, pilot warehouse offering for a small number of customers.
- The new design is a Trino-based distributed query engine over a custom table format (“HogLake”), intended mainly for long-running modeling and ETL, not low‑latency queries.
DuckDB experience and scaling limits
- DuckDB worked functionally but required very large pods (on the order of ~46 vCPUs and hundreds of GB RAM) to handle real customer queries, some running 12+ hours.
- Under multi‑tenant load (many heavy queries from the same customer), this became economically unviable.
- Estimating per‑query CPU/memory usage proved hard. A DuckDB client/server protocol (Quack) was noted as not supporting true distributed querying.
Rationale for Trino + lake-style architecture
- Trino is seen as offering more flexible, distributed resource allocation, better matching bursty, heavy workloads.
- DuckDB is praised as an excellent single‑node engine but viewed as a mismatch for this cloud, multi‑tenant context.
- Iceberg/Delta were evaluated but rejected due to slow writes and other downsides for this use case.
Critiques: planning, churn, and methodology
- Many commenters see switching from ClickHouse → DuckDB → Trino/HogLake within months as poor planning or “churn,” with insufficient up‑front scale testing.
- Some frame it as resume‑driven or “agent/LLM‑driven” development, swapping technologies instead of deeply tuning one.
- Others defend experimentation, arguing real behavior at scale often only emerges under organic workloads.
Desire for more technical depth and community engagement
- Multiple commenters wanted concrete metrics, failure modes, and methodology for capacity prediction and testing, plus clearer explanation of why a custom format was needed.
- Some argue maintaining forks and upstreaming improvements is normal at scale and wanted more detail on patches and collaboration.
- It remains unclear from the discussion whether the new Trino/HogLake setup has definitively solved the original issues.
Miscellaneous reactions
- Some criticize the company’s marketing blog style and website design; others find them distinctive or appealing.
- Broader themes raised include the difficulty of database choices without deep expertise, leadership pattern‑matching failures, and the risk of frequent foundational rewrites.