1egg0myegg0

9 ms·

61.
by 1egg0myegg0 4y ago
DuckDB is not only an in memory DB - it can persist data and handle larger than RAM workloads. We are expanding the number of operators that can handle larger than memory data, but it is a key design goal!
62.
by 1egg0myegg0 4y ago
DuckDB can certainly handle larger than memory data! It's a streaming engine, so it doesn't need everything in memory all at once. Some operators (like sorting) can also buffer out to disk if you run low on memory mid-computation.
63.
by 1egg0myegg0 5y ago
Since DuckDB is so portable, it can be a good choice for adding a SQL parser to a new system. Have a look here for an example that uses the parser separately from the execution engine: https://github.com/duckdb/duckdb&#
64.
by 1egg0myegg0 5y ago
It is just scanning the data (in parallel), not copying it into DuckDB!
65.
by 1egg0myegg0 5y ago
Does this support live updates as other users make edits (without refreshing the page)?
66.
by 1egg0myegg0 5y ago
If you wouldn't mind, please update DuckDB as well!
67.
by 1egg0myegg0 5y ago
Polars would be 10-100x faster, but so would DuckDB!
68.
by 1egg0myegg0 5y ago
Have you thought about interfacing with DuckDB for out of core processing? https://github.com/kimmolinna/DuckDB.jl
69.
by 1egg0myegg0 5y ago
I should also add that there is a duckdb fdw, so you could have DuckDB read from your parquet files and do faster transformations before you pull your data into Postgres! https://github.com/alitrack/duckdb_fdw
70.
by 1egg0myegg0 5y ago
I am not very familiar with the Postgres Parquet FDW, but here is an educated guess! Postgres is a row store engine rather than a column store, so I believe there will need to be quite a lot of translation for Postgres to be able to process
71.
by 1egg0myegg0 5y ago
DuckDB has a few options for custom functions! I'm not sure exactly how optimized they are relative to what you have in mind, but I believe there is still quite a lot of query optimization that you get out of the box. The first option
72.
by 1egg0myegg0 5y ago
Yes! DuckDB (not WASM DuckDB) has a JDBC connector that works with DBeaver.
73.
by 1egg0myegg0 5y ago
If you head over to the shell demo, you can run a query like the one below! https://shell.duckdb.org/ select * from ' https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parqu... &
74.
by 1egg0myegg0 5y ago
NIIIICE! Data twitter was pretty excited about that cool SQLite trick - now you can turn it up a notch!
75.
by 1egg0myegg0 5y ago
Check out this post for some comparisons with Pandas. https://duckdb.org/2021/05/14/sql-on-pandas.html DuckDB is often faster than Pandas, and it can handle larger than memory data. Plus, if you already know