3 ms·
Did you consider Clickhouse? join's are slow, but if your data is in a single table, it works really well.
by mritchie712 4y ago
Did you consider Clickhouse? join's are slow, but if your data is in a single table, it works really well.
- marcinzm 4y agoWe were trying to keep everything on one machine in (mostly) memory for simplicity. Once you open up the pandoras box of distributed compute there's a lot of options including other ways of running Spark. But yes, in retrospect, we should have opened that box first.
- anko 4y agoI have solved a similar problem, in a similar way and i've found polars <https://www.pola.rs/ https://www.pola.rs/> to solve this quite well without needing clickhouse. It has a python library but does most processing in rust, across multiple cores. I've used it for data sets up to about 20GB no worries, but my computer's ram became the issue, not polars itself.
- marcinzm 4y agoWe were using 500+gb of memory at peak and were expecting that to grow. If I remember we didn't go with Polars because we needed to run custom apply functions on DataFrames. Polars had them but the function took a tuple (not a DF or dict) which when you've got 20+ columns makes for really error prone code. Dask and Spark both supported a batch transform operation so the function took a Pandas Dataframe as input and output.