4 ms·
We’re currently building real-time apis backed by terabytes of compressed parquet… hundreds of billions of ‘rows’… in exactly this fashion using polars. It ama
by jmartin2683 4y ago
We’re currently building real-time apis backed by terabytes of compressed parquet… hundreds of billions of ‘rows’… in exactly this fashion using polars. It amazes us at every turn.
Join us and help!
- ploppyploppy 4y agoWhat project? Do you mean polars reading Parquet into DuckDB to process that amount of data?
- jmartin2683 4y agoInternal. We're using Polars as the query engine to effectively query that data statically at rest (more accurately, mmap'd on disk in arrow ipc format)