3 ms·
well, pg_mooncake is a Postgres extension, and Postgres + pg_mooncake is still just Postgres. Users deploy pg_mooncake as a Postgres extension and write and que
by moonikakiss 2y ago
well, pg_mooncake is a Postgres extension, and Postgres + pg_mooncake is still just Postgres. Users deploy pg_mooncake as a Postgres extension and write and query all tables through psql.
Fast analytic databases need two key things: columnar storage and a vectorized execution engine. We introduce a columnstore table access method in Postgres with data stored in Parquet) and execute queries on those tables using DuckDB.
By leveraging DuckDB's execution engine, we avoid reinventing vectorized query execution while keeping everything managed through Postgres.
More on our architecture: https://www.mooncake.dev/blog/how-we-built-pgmooncake https://www.mooncake.dev/blog/how-we-built-pgmooncake
- moonikakiss 2y agoOne thing I forgot to mention. We're actually faster than DuckDB on parquet. We implemented segment elimination based on parquet metadata. We blogged about this: https://www.mooncake.dev/blog/duckdb-parquet https://www.mooncake.dev/blog/duckdb-parquet
- porker 2y agoHave you upstreamed the change to DuckDB?
- fmbb 2y agoReading the linked blog it seems the speed up comes from offloading some lookups to Postgres tables. That doesn’t seem like something that makes sense to in DuckDB. > pg_mooncake leverages Postgres to store table metadata, eliminating the need to query an external catalog (as DuckDB would require when scanning external Iceberg or Delta Lake tables). Instead, it directly queries the Postgres table to identify which Parquet files to scan, reducing overhead. > Additionally, pg_mooncake stores detailed Parquet metadata for each file in Postgres, including column statistics for every row group. These statistics, such as MIN/MAX values for each column, are aggregated to optimize scans. When querying a columnstore table, pg_mooncake uses these statistics to filter out data files and row groups that cannot produce matches.
- fifilura 2y agoWhat is the cost of scaling up number of CPUs for parallel processing? That was always the culprit for me compared to AWS/Athena and BigQuery. They are dirt cheap on analytics workloads when you can parallelize the calculations to 100 CPUs without really paying any extra. With postgres you are stuck with linear cost for scaling up number of CPUs, so everything is slow anyway.
- zhousun 2y agozhou from mooncake labs here. Good point! Normally for postgres extension it won't be solvable, but for mooncake it is actually not the case! The core idea of mooncake is to built upon open columnar format + substitutable vectorized engine, while natively integrate with Postgres. So right now it is using duckdb within postgres to run the query, but we can and we will support ad-hoc using other 'stateless engines' like Athena, StarRocks or even spark to run a big query.
- gavinray 2y agoCan I suggest you put this sentence somewhere in the README near the beginning? > Fast analytic databases need two key things: columnar storage and a vectorized execution engine. We introduce a columnstore table access method in Postgres with data stored in Parquet) and execute queries on those tables using DuckDB.
- jbergens 2y agoI'm personally on the side that thinks that pg + x != pg. It might not even be possible to install x on some hosted pg servers. It is great that you can expand pg and that you and others have but I don't rate it as high as a buil-in solution.
- nextn 2y agoFast analytics databases need a third key technology. I can't say what. You seem unaware of it.