2 ms·
Spark feels overwrought, but polars does not scale out yet. https://github.com/pola-rs/polars/issues/5621 https://github.com/pola-rs/polars/issues/5621
by esafak 3y ago
Spark feels overwrought, but polars does not scale out yet.
https://github.com/pola-rs/polars/issues/5621 https://github.com/pola-rs/polars/issues/5621
- appplication 3y agoInteresting, that is a huge nonstarter for us then. I always thought of polars as “pandas but distributed” but I see I have been incorrect in that assessment.
- theLiminator 3y agoYou might want to consider seeing how far vertical scaling can bring you. https://motherduck.com/blog/big-data-is-dead/ https://motherduck.com/blog/big-data-is-dead/ Is a good read
- appplication 3y agoThat is a good read, and I see most of those points for probably even most midsize companies. I think in this case we are data 1%-ers. We generate and process terabytes of data every day, so the need for horizontal scaling is real.
- sanderjd 3y agoNot super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista https://github.com/apache/arrow-ballista