3 ms·
I think that neither "outlier toolchain" nor "some percentage increase" are fair. This benchmark [0] show significant speedup while lowering the memory needs. Y
by wdroz 4y ago
I think that neither "outlier toolchain" nor "some percentage increase" are fair. This benchmark [0] show significant speedup while lowering the memory needs. You still need to reach dask/spark for really big data where you need a cluster of beefy computers for your tasks.
If you use an r5d.24xlarge-like[1] instance, you can skip spark/dask for most workflows as 768 GB is plenty enough. On top of that, polars will efficiently use the 96 available cores when you are computing your join, groupby, etc.
Also polars is getting more and more popular[2]
[0] -- https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
[1] -- https://aws.amazon.com/fr/ec2/instance-types/c6a/ https://aws.amazon.com/fr/ec2/instance-types/c6a/
[2] -- https://star-history.com/#pola-rs/polars&Date https://star-history.com/#pola-rs/polars&Date