4 ms·
I wish if there's a way we can hook DuckDB as Extension to Postgres itself. Currently there's no proper columnar database/extension exist in PG. In most of the
by nitinreddy88 3y ago
I wish if there's a way we can hook DuckDB as Extension to Postgres itself.
Currently there's no proper columnar database/extension exist in PG. In most of the use cases, one doesn't really require dedicated "analytics" only DB and sometimes everything has to be done through "Primary DB" (typically in enterprise products world).
By hooking DuckDB as Extension, we get best of both of worlds from PG and DuckDB while retaining the persistence and analytical capabilities without hosting yet another DB infrastructure.
- tehlike 3y ago+1. I checked Citus, hydra, etc, and nothing comes close, all has its restrictions or gotchas. I also really want to have a LSM based storage engine, but noone implemented it yet. Yugabyte built a postgres compatible database, but it has some management-related stuff not opensource... Cockroach similarly...
- philippemnoel 3y agoCheck out https://github.com/paradedb/paradedb/tree/dev/pg_analytics https://github.com/paradedb/paradedb/tree/dev/pg_analytics, we're shipping this week for column-based storage in Postgres. It's not LSM, though, but Parquet/Arrow
- wenc 3y agoSeems like someone has tried it: https://medium.com/@ahuarte/loading-parquet-in-postgresql-via-duckdb-testing-queries-and-exploring-the-core-1d667ae67dc2 https://medium.com/@ahuarte/loading-parquet-in-postgresql-vi...
- nitinreddy88 3y agoNow we need to write parquet files from Postgres. Its essentially what do we need to do for end to end solution with PG by leveraging extensions
- tehlike 3y agohttps://github.com/adjust/parquet_fdw https://github.com/adjust/parquet_fdw
- nitinreddy88 3y agoIt doesn't support writing data. One has to hookup worker to write data efficiently to parquet on regular basis or some fashion using PG. Its like bits and pieces everywhere but didnt create end to end usecase
- ibotty 3y agoThis can be done using duckdb_fdw. It needs one additional copy though. You insert into the duckdb table and then export from duckdb using duckdb_execute.
- nattaylor 3y agopg_analytics looks promising here https://github.com/paradedb/paradedb/tree/dev/pg_analytics https://github.com/paradedb/paradedb/tree/dev/pg_analytics
- ako 3y agoWouldn’t you still miss the part about storing tables column based in your database rather than row based? Would be nice to have “create table” or “create materialized” view with columnar storage option.
- philippemnoel 3y agoHey! I'm one of the makers of pg_analytics/ParadeDB. We store tables column-based. You can do CREATE TABLE (...) USING deltalake; when using ParadeDB and it will be stored column-based (and soon, with support for also querying cloud object stores!)
- ako 3y agoLooks interesting, but exactly what is missing when just adding duckdb to Postgres. Why did you go with delta lake over iceberg?
- philippemnoel 3y agoDuckDB is in-memory. It makes sense as a in-memory columnar cache. We went with DataFusion to benefit from the deep integration it has into the ecosystem with Arrow, Parquet, etc. As for Delta Lake, we would've loved to do Iceberg but the Rust crate for Iceberg is very new, and doesn't have enough functionalities yet. We're tracking the project and planning to contribute, so that we can add support for Iceberg eventually