4 ms·

Been watching polars for a while. Look promising, would love to test it against spark some day. So much of our infra is on spark though so we’re probably just l

• esafak 3y ago
  Spark feels overwrought, but polars does not scale out yet. https://github.com/pola-rs/polars/issues/5621 https://github.com/pola-rs/polars/issues/5621
  • appplication 3y ago
    Interesting, that is a huge nonstarter for us then. I always thought of polars as “pandas but distributed” but I see I have been incorrect in that assessment.
    • theLiminator 3y ago
      You might want to consider seeing how far vertical scaling can bring you. https://motherduck.com/blog/big-data-is-dead/ https://motherduck.com/blog/big-data-is-dead/ Is a good read
      • appplication 3y ago
        That is a good read, and I see most of those points for probably even most midsize companies. I think in this case we are data 1%-ers. We generate and process terabytes of data every day, so the need for horizontal scaling is real.
  • sanderjd 3y ago
    Not super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista https://github.com/apache/arrow-ballista