PostHog moves its data warehouse away from DuckDB

Five months after rebuilding its data warehouse on DuckDB instead of ClickHouse, PostHog has announced a complete architectural pivot. The company moved away from DuckDB due to the unexpected complexity of customer workloads and the limitations of running single-node engines in the cloud.

• PostHog migrated its warehouse to DuckDB five months ago to handle complex modeling queries.
• The single-node engine faced scaling limitations and high complexity in cloud environments.
• DuckLake held around 2 petabytes of data, roughly 1 trillion rows, and 15.3 million snapshots.
• The company re-architected its infrastructure around a custom flavor of Trino and HogLake.

Challenges of single-node engines in the cloud

While DuckDB is widely regarded as an exceptional single-node tool for local analytics, running it safely in the cloud presented unique ergonomic hurdles. Customers executed far more complex queries than anticipated, forcing the team to provision massive instances. Predicting CPU and memory requirements for specific user queries proved extremely difficult.

Users should not have to worry about resource provisioning or sizing worker nodes beforehand. Additionally, fetching large amounts of data from S3 without native async fetching created bottlenecks that slowed down overall processing times and increased operational overhead.

DuckLake scalability and catalog bottlenecks

PostHog adopted the DuckLake extension to run DuckDB as a fully-fledged data warehouse, but it hit major limits at petabyte scale. Because DuckLake's catalog schema must remain portable across multiple SQL databases, engine-specific optimizations were severely restricted. A single-table write could take over three minutes because every commit re-read catalog statistics.

Over 99% of the catalog's statistics rows belonged to deleted tables, and maintaining compatibility with the official schema left engineers with fewer optimization paths. Sticking to an internal fork defeated the collaborative benefits of the open-source community, prompting the team to rethink their storage layer entirely.

Difficulties with Postgres wire compatibility

Exposing direct Postgres connections over port 5432 seemed like a great feature, but it proved far harder to scale than standard HTTP. Managing idle timeouts, handling dropped connections, and load-balancing pgwire lacked the mature tooling available for HTTP protocols in Kubernetes environments.

An endless variety of Postgres clients with niche behaviors meant PostHog spent significant engineering time fixing client-specific compatibility issues rather than focusing on building a superior core warehouse product.

What comes next for PostHog

The migration taught the team not to over-hype the underlying database technology, but rather focus on delivering a seamless user experience. PostHog rebuilt its entire stack around a custom flavor of Trino and introduced a brand-new open table format called HogLake, thoroughly tested at petabyte scale before rollout.