4 ms·
When pandas can’t manage a data set, I would convert to hdf5 and use the vaex pandas-like API. This is very fast and memory efficient (out of core) for my typic
by blub 4y ago
When pandas can’t manage a data set, I would convert to hdf5 and use the vaex pandas-like API. This is very fast and memory efficient (out of core) for my typical 40-70GB data set size and reportedly remains fast up to 1 TB.
The key ingredient is the on-disk format: hdf5 and more recently also arrow.
Looks like DuckDb would force me to use SQL, which is not what I typically want. That being said, if this offered better data management than a binary blob like hdf5, it might be worthwhile. If the use case requires processing historical data for example, a real DB would be better than juggling 10 x 40GB files or having one big 400GB file that keeps growing.
- 1egg0myegg0 4y agoWe have a Relational API in addition to SQL! Here are some examples for the Python Relational API client: https://github.com/duckdb/duckdb/blob/master/examples/python/duckdb-python.py https://github.com/duckdb/duckdb/blob/master/examples/python... Plus, if you are working in Python, you can use DuckDB as the engine underneath Ibis, Fugue, Siuba, or anything that works with SQLAlchemy (using the DuckDB-engine driver)! In R, you can use dplyr or dbplyr. DuckDB's file format is one way to persist data (it uses a single file), but you can also write out to Parquet, or write out to Apache Arrow and then parquet (in a partitioned format I believe). Disclaimer - I write docs for DuckDB!