4 ms·
Is the sole appeal of polars (vs, say, pandas) its execution speed? I've found being able to express ideas clearly in code (to aid comprehension now and in the
by nomilk 3y ago
Is the sole appeal of polars (vs, say, pandas) its execution speed?
I've found being able to express ideas clearly in code (to aid comprehension now and in the future) to be much more important than shaving off a few seconds of run time.
For this reason I think speed alone is not a strong sell point, except specifically in cases where execution times really matter.
Analogous somewhat to how ruby/rails might be a 'slow' language/framework (e.g. 600ms when another framework might be 200ms) but multiples faster in facilitating the expression of complex ideas through code, which tends to be the far bigger problem in most software projects.
- singhrac 3y agoThey're similarly expressive, and there are some areas where Polars (taking ideas from years of Pandas development) is ahead, for example with nullable dtypes. That being said, there are many, many times when I would be willing to rewrite code to make it faster or more memory efficient. Just yesterday I rewrote a method from Pandas to Polars to take advantage of that. Time might be cheap (while developing) but memory is expensive. Similarly, if you're writing data science jobs for production, you care about both.
- DanielVZ 3y agoIt strives for a more consistent Dataframe API too. It’s quite subjective but I prefer it
- nerdponx 3y agoThe advantages are: raw performance, an optimizing query engine, streaming/out-of-memory processing support, backed by Arrow so you can load data with zero copy. Some people also prefer the API, which is very similar to that of Spark SQL and might feel more comfortable and consistent to people with professional programming backgrounds.
- sanderjd 3y agoPersonally I prefer its API as well, but that seems to be a more controversial opinion than its often-huge performance wins.
- maronato 3y agoComplex operations on very large datasets can take multiple minutes in pandas. Polars is supposed to reduce that to a few seconds. Since a lot of people use pandas to explore and experiment with datasets, having your workflow speed limited to a few operations an hour is hard to defend. That’s where the value proposition of polars and similar solutions lie IMO.
- nomilk 3y ago> operations on very large datasets can take multiple minutes in pandas. Polars is supposed to reduce that to a few seconds. Good example; I can see how efficiency would matter for workflows like that. I work with dataframes in the 10's-100's millions of rows (mostly in tidyverse, but also pandas and base python and R), and find most data wrangling operations are close to instant on modern laptops. Plotting is another story (not sure if polars helps there). So the case for efficiency is weak at the 10-100 million row dataframe size (unless doing some intense computations), but gains strength as the size of the dataframe grows. Would be a fun aside to test all these frameworks side by side with some 1m/10m/100m/1bn row joins, filters, summary calcs, maps etc to get some concrete data on where efficiency starts to become noticeable and starts to matter. I think at sub 100m rows it probably doesn't. Not for the kinds of operations I do anyway.
- dash2 3y agoI'd be interested to know what proportion of users of dataframes are working at different orders of magnitude. Most of my life I've had databases of like 1000. Now I have a big one of about 500K! So for me, speed is almost a non-issue. But that is my specific field.
- antonvs 3y agoThere are lots of cases where execution times really matter. We use polars for exactly that reason.