3 ms·
1) and 2) Backups and replication are on the roadmap. It's the next major feature we're working on. 3) We store data in Parquet files, which is a heavily compr
by philippemnoel 3y ago
1) and 2) Backups and replication are on the roadmap. It's the next major feature we're working on.
3) We store data in Parquet files, which is a heavily compressed file format for columnar data. This ensures good compression and compatibility with Arrow for in-memory columnar processing. We also hook at the executor level and route queries on deltalake tables to DataFusion query engine, which processes the data in a vectorized fashion for much faster execution
- pgaddict 3y agoThanks. I wonder how you plan to replicate stuff, considering how heavily it relies on WAL (I haven't found the answer in the code, but I'm not very familiar with rust). How large part of the plan you route to the datalake tables? Just scans or some more complex part? Can you point me to the part of the code doing that? I'm intrigued.
- philippemnoel 3y agoHey! Sorry I missed this. 1) You can find the code here: https://github.com/paradedb/paradedb/blob/996f018e3258d3989f2020a3c98acf10a0a37c42/pg_analytics/src/hooks/executor.rs#L4 https://github.com/paradedb/paradedb/blob/996f018e3258d3989f... For deltalake tables, we send the full plan to DataFusion. 2) Re: WALs and replication -- We are currently adding support for WALs