PostHog, a product analytics and data infrastructure platform, recently announced a significant architectural shift in its data warehouse. After initially migrating from ClickHouse to DuckDB to solve multi-tenancy issues and gain access to a cost-based query optimizer, the company is now moving away from DuckDB. The move is driven by the need to support more complex, large-scale workloads that exceeded the practical limits of a single-node engine in a cloud environment.
This image serves as the primary visual for the article's core topic: the transition from a single-node DuckDB setup to a distributed architecture.
While DuckDB is highly effective for local analysis and single-node queries, PostHog found that scaling it for enterprise-level workloads presented significant hurdles. Because DuckDB is a single-node tool, the company had to size worker instances before queries arrived, a process that proved difficult for predicting the CPU and memory requirements of unpredictable customer queries. Furthermore, while DuckDB can spill to disk when memory is exhausted, this resulted in queries that were too slow and expensive to be viable for their users. The company also encountered bottlenecks with the DuckLake extension, which was used to maintain compatibility with standard lakehouse formats. At PostHog's scale—which includes roughly 1 trillion rows and 2 petabytes of data—DuckLake's catalog schema became a bottleneck, causing single-table writes to take over three minutes because every commit required re-reading statistics for the entire catalog.
To address these limitations, PostHog is rebuilding its warehouse using Trino, a distributed SQL query engine, paired with a custom open table format called HogLake. This shift aims to provide a distributed architecture that can handle petabyte-scale data more predictably than a single-node setup. Additionally, the move addresses the difficulties of maintaining a Postgres Wire protocol for tool compatibility; PostHog found that managing long-running Postgres connections and the diverse requirements of various clients was harder to scale than using HTTP-based routing. While the new Trino-based architecture is currently being tested at PostHog's scale, the company has not yet released specific performance benchmarks for the new system compared to the previous DuckDB implementation.