3 ms·
One thing I forgot to mention. We're actually faster than DuckDB on parquet. We implemented segment elimination based on parquet metadata. We blogged about thi
by moonikakiss 2y ago
One thing I forgot to mention. We're actually faster than DuckDB on parquet. We implemented segment elimination based on parquet metadata.
We blogged about this: https://www.mooncake.dev/blog/duckdb-parquet https://www.mooncake.dev/blog/duckdb-parquet
- porker 2y agoHave you upstreamed the change to DuckDB?
- fmbb 2y agoReading the linked blog it seems the speed up comes from offloading some lookups to Postgres tables. That doesn’t seem like something that makes sense to in DuckDB. > pg_mooncake leverages Postgres to store table metadata, eliminating the need to query an external catalog (as DuckDB would require when scanning external Iceberg or Delta Lake tables). Instead, it directly queries the Postgres table to identify which Parquet files to scan, reducing overhead. > Additionally, pg_mooncake stores detailed Parquet metadata for each file in Postgres, including column statistics for every row group. These statistics, such as MIN/MAX values for each column, are aggregated to optimize scans. When querying a columnstore table, pg_mooncake uses these statistics to filter out data files and row groups that cannot produce matches.