5 ms·
Requirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet
by jmartin2683 4y ago
Requirements largely require selecting from and aggregating across some subset of the whole universe of data, which is on the order of hundreds of GB of parquet. We use partitioning across files (and folder structure in s3) to lazily scan only the bits of data we need, then predicate and projection push down (thanks to polars) to reduce that further.
- Kinrany 4y agoThanks. If you don't mind, what's the size of your individual files, and how many of them does it take to serve a single request: one, a few, N?
- jmartin2683 4y agoVariable, but a single partitioned file goes anywhere from single-digit kb to maybe a little under a meg. A query can touch just one or dozens (all lazily concatenated into a single frame).