4 ms·
There is a new open source columnar lakehouse gaining traction with low cost data processing and storage, with Polars as part of it: 1. Querying and Data proce
by jamesblonde 3y ago
There is a new open source columnar lakehouse gaining traction with low cost data processing and storage, with Polars as part of it:
1. Querying and Data processing in Polars or DuckDB
2. Metadata (for transactions, time-travel) table formats (Iceberg, Hudi, Delta)
3. Storage with Parquet on S3.
With Polars/DuckDB, you can process up to 100+ GB of data at a time on a single VM, turn it off when you're not using it. It's a lower overhead, lower time-to-value stack than Spark. And even if you have TBs of data, so long as you only process ~100GB at a time, it works fine.
- tsss 3y agoWhy the 100GB hard limit? If you can stream from disk you should be able to process infinite data sets if the operations allow it. If you need to do stuff like deduplication, then it will depend on how much RAM you have available.
- jamesblonde 3y agoThere's no "limit", just a kind of practical size I have seen. It's a single VM. You will need NVMe to prevent disk being a bottleneck (NVMe is now 8 GB/s vs 200 MB/s for S3).
- DEDLINE 3y agoKnow of any papers / articles on this stack?