8 ms·
The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never c
by edschofield 8mo ago
The design of Pandas is inferior in every way to Polars: API, memory use, speed, expressiveness. Pandas has been strictly worse since late 2023 and will never close the gap. Polars is multithreaded by default, written in a low-level language, has a powerful query engine, supports lazy, out-of memory execution, and isn’t constrained by any compatibility concerns with a warty, eager-only API and pre-Arrow data types that aren’t nullable.
It’s probably not worth incurring the pain of a compatibility-breaking Pandas upgrade. Switch to Polars instead for new projects and you won’t look back.
- v3ss0n 8mo agoSounds too much like an advertisement. Also we need to watch out when diving into Polars . Polars is VC backed Opensource project with cloud offering , which may become an opencore project - we know how those goes.
- gkbrk 8mo ago> we know how those go They get forked and stay open source? At least this is what happens to all the popular ones. You can't really un-open-source a project if users want to keep it open-source.
- stingraycharles 8mo agoDepends on your definition of popular; plenty of examples where the business interests don't align well with open source.
- v3ss0n 8mo agonot many can maintain a complex project in full time.
- quentindanjou 8mo agoI was also thinking that this comment looks like an AD. Pandas does not have any paid option and isn't made directly for profit.
- disgruntledphd2 8mo agoTo be fair, as someone who's fought pandas for many years I agree with basically everything they said. The API design for Polars is much, much more intuitive. It's a base R to dplyr level change.
- rich_sasha 8mo agoI almost fully agree. I would add that Pandas API is poorly thought through and full of footguns. Where I certainly disagree is the "frame as a dict of time series" setting, and general time series analysis. The feel is also different. Pandas is an interactive data analysis container, poorly suited for production use. Polars I feel is the other way round.
- sirfz 8mo agoI think that's a sane take. Indeed, I think most data analysts find it much easier to use pandas over polars when playing with data (mainly the bracket syntax is faster and mostly sensible)
- thelastbender12 8mo agoI think that's a fair opinion, but I'd argue against it being poorly thought out - pandas HAS to stick with older api decisions (dating back to before data science was a mature enough field, and it has pandas to thank for much of it) for backwards compatibility.
- ohyoutravel 8mo agoWell this is like saying Python must maintain backwards compatibility with Python 2 primitives for all time. It’s simply not true. It’s not easy to deprecate an old API, but it’s doable and there are playbooks for it. Pandas is good, I’ve used it extensively, but agree it’s not fit for production use. They could catch up to the state of the art, but that requires them being very opinionated and willing to make some unpopular decisions for the greater good.
- cruffle_duffle 8mo agoWhy though? polars sounds like the rewrite! It’s okay to cycle into a new library. Let pandas do its thing and polars slowly take over as new projects overtake. There is nothing wrong with this and it happens all the time. Like jquery, which hasn’t fundamentally changed since I was a wee lad doing web dev. They didn’t make major changes despite their approach to web dev being replaced by newer concepts found on angular, backbone, mustache, and eventually react. And that is a good thing. What I personally don’t want is something like angular that basically radically changed between 1.0 and 2.0. Might as well just call 2.0 something new. Note: I’ve never heard of polars until this comment thread. Can’t wait to try it out.
- deleted 8mo ago[deleted]
- lairv 8mo agoI would agree if not for the fact that polars is not compatible with Python multiprocessing when using the default fork method, the following script hangs forever (the pandas equivalent runs): import polars as pl from concurrent.futures import ProcessPoolExecutor pl.DataFrame({"a": [1,2,3], "b": [4,5,6]}).write_parquet("test.parquet") def read_parquet(): x = pl.read_parquet("test.parquet") print(x.shape) with ProcessPoolExecutor() as executor: futures = [executor.submit(read_parquet) for _ in range(100)] r = [f.result() for f in futures] Using thread pool or "spawn" start method works but it makes polars a pain to use inside e.g. PyTorch dataloader
- schmidtleonard 8mo agoI can't believe parallel processing is still this big of a dumpster fire in python 20 years after multi-core became the rule rather than the exception. Do they really still not have a good mechanism to toss a flag on a for loop to capture embarrassing parallelism easily?
- skylurk 8mo agoYou are not wrong, but for this example you can do something like this to run in threads: import polars as pl pl.DataFrame({"a": [1, 2, 3]}).write_parquet("test.parquet") def print_shape(df: pl.DataFrame) -> pl.DataFrame: print(df.shape) return df lazy_frames = [ pl.scan_parquet("test.parquet") .map_batches(print_shape) for _ in range(100) ] pl.collect_all(lazy_frames, comm_subplan_elim=False) (comm_subplan_elim is important)
- bhadass 8mo agowhy not just go full bore to duckdb?
- vegabook 8mo agobecause method chaining in Polars is much more composable and ergonomic than SQL once the pipeline gets complex which makes it superior in an exploratory "data wrangling" environment.
- data-ottawa 8mo agoDuckdb does support pipe operators as an extension, which is a welcome addition to sql engines for me. But I do agree with you.
- data-ottawa 8mo agoA dataframe API allows you to write code in Python, with native syntax highlighting and your LSP can complete it, in one analysis file. Inlined SQL is not as nice, and has weird ergonomics. UDFs in most dataframe libraries tend to feel better than writing udfs for a sql engine as well. Polars specifically has lazy mode which enables a query optimizer, so you get predicate push down and all the goodies if SQL, with extra control/primitives (sane pivoting, group_by_dynamic, etc) I do use ibis on top of duckdb sometimes, but the UDF situation persists and the way they organize their docs is very difficult to use.
- sampo 8mo agoHistorically 18 years ago, Pandas started as a project by someone working in finance to use Python instead of Excel, yet be nicer than using just raw Python dicts and Numpy arrays. For better or worse, like Excel and like the simpler programming languages of old, Pandas lets you overwrite data in place. Prepare some data df_pandas = pd.DataFrame({'a': [1, 2, 3, 4, 5], 'b': [10, 20, 30, 40, 50]}) df_polars = pl.from_pandas(df_pandas) And then df_pandas.loc[1:3, 'b'] += 1 df_pandas a b 0 1 10 1 2 21 2 3 31 3 4 41 4 5 50 Polars comes from a more modern data engineering philosopy, and data is immutable. In Polars, if you ever wanted to do such a thing, you'd write a pipeline to process and replace the whole column. df_polars = df_polars.with_columns( pl.when(pl.int_range(0, pl.len()).is_between(1, 3)) .then(pl.col("b") + 1) .otherwise(pl.col("b")) .alias("b") ) If you are just interactively playing around with your data, and want to do it in Python and not in Excel or R, Pandas might still hit the spot. Or use Polars, and if need be then temporarily convert the data to Pandas or even to a Numpy array, manipulate, and then convert back. P.S. Polars has an optimization to overwite a single value df_polars[4, 'b'] += 5 df_polars ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 1 ┆ 10 │ │ 2 ┆ 21 │ │ 3 ┆ 31 │ │ 4 ┆ 41 │ │ 5 ┆ 55 │ └─────┴─────┘ But as far as I know, it doesn't allow slicing or anything.
- goatlover 8mo agoThe Polars code puts me off as being too verbose and requiring too many steps. I love the broadcasting ability that Pandas gets from Numpy. It's what sceintific computing should look like in my opinon. Maybe R, Julia or some array-based language does it a bit better than Numpy/Pandas, but it's certainly not like the Polars example.
- thereisnospork 8mo agoLikewise, I was considering trying Polaris until I saw that example. The pandas example is a good approximation of how I think and want to transform/process data even if it is ugly under the hood. I do occasionally find numpy and pandas annoying wrt when the return a view vs a copy but the cure seems worse than the disease.
- noo_u 8mo agoPolars took a lot of ideas from Pandas and made them better - calling it "inferior in every way" is all sorts of disrespectful :P Unfortunately, there are a lot of third party libraries that work with Pandas that do not work with Polars, so the switch, even for new projects, should be done with that in mind.
- skylurk 8mo agoLuckily, polars has .to_pandas() so you can still pass pandas dataframes to the libraries that really are still stuck on that interface. I maintain one of those libraries and everything is polars internally.
- deleted 8mo ago[deleted]
- satvikpendem 8mo ago"If I have seen further, it is by standing on the shoulders of giants" - Isaac Newton Polars is great, but it is better precisely because it learned from all the mistakes of Pandas. Don't besmirch the latter just because it now has to deal with the backwards compatibility of those mistakes, because when it first started, it was revolutionary.
- Xunjin 8mo agoIndeed, even Rust was created learning with the mistakes of memory management and known patterns like the famous RAII.
- vegabook 8mo ago"revolutionary"? It just copied and pasted the decades-old R (previous "S") dataframe into Python, including all the paradigms (with worse ergonomics since it's not baked into the language).
- data-ottawa 8mo agoNo other modern language will compete with R on ergonomics because of how it allows functions to read the context they’re called in, and S expressions are incredibly flexibly. The R manual is great. To say pandas just copied it but worse is overly dismissive. The core of pandas has always been indexing/reindexing, split-apply-combine, and slicing views. It’s a different approach than R’s data tables or frames.
- aidos 8mo ago> allows functions to read the context they’re called in Can you show an example? Seems interesting considering that code knowing about external context is not generally a good pattern when it comes to maintainability (security, readability). I’ve lived through some horrific 10M line coldfusion codebases that embraced this paradigm to death - they were a whole other extreme where you could _write_ variables in the scope of where you were called from!
- rdedev 8mo agoWhile polars is better if you work with predefined data formats, pandas is imo still better as a general purpose table container. I work with chemical datasets and this always involves converting SMILES string to Rdkit Molecule objects. Polars cannot do this as simply as calling .map on pandas. Pandas is also much better to do EDA. So calling it worse in every instance is not true. If you are doing pure data manipulation then go ahead with polars
- data-ottawa 8mo agoMap is one operation pandas does nicely that most other “wrap a fast language” dataframe tools do poorly. When it feels like you’re writing some external udf thats executed in another environment, it does not feel as nice as throwing in a lambda, even if the lambda is not ideal.
- vegabook 8mo agoyou have map_elements in polars which does exactly this. https://docs.pola.rs/api/python/dev/reference/expressions/api/polars.Expr.map_elements.html https://docs.pola.rs/api/python/dev/reference/expressions/ap... You can also iter_rows into a lambda if you really want to. https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.iter_rows.html https://docs.pola.rs/api/python/stable/reference/dataframe/a... Personally I find it extremely rare that I need to do this given Polars expressions are so comprehensive, including when.then.otherwise when all else fails.
- data-ottawa 8mo agoThat one has a bit more friction than pandas because the return schema requirement -- pandas let's you get away with this bad practice. It also does batches when you declare scalar outputs, but you can't control the batch size, which usually isn't an issue, but I've run into situations where it is.
- deleted 8mo ago[deleted]
- data-ottawa 8mo agoPandas deserves a ton of respect in my opinion. I built my career on knowing it well and using it daily for a decade, so I’m biased. Pandas created the modern Python data stack when there was not really any alternatives (except R and closed source). The original split-apply-combine paradigm was well thought out, simple, and effective, and the built in tools to read pretty much anything (including all of your awful csv files and excel tables) and deal with timestamps easily made it fit into tons of workflows. It pioneered a lot, and basically still serves as the foundation and common format for the industry. I always recommend every member of my teams read Modern Pandas by Tom Augspurger when they start, as it covers all the modern concepts you need to get data work done fast and with high quality. The concepts carry over to polars. And I have to thank the pandas team for being a very open and collaborative bunch. They’re humble and smart people, and every PR or issue I’ve interacted with them on has been great. Polars is undeniably great software, it’s my standard tool today. But they did benefit from the failures and hard edges of pandas, pyspark, dask, the tidyverse, and xarray. It’s an advantage pandas didn’t have, and they still pay for. I’m not trying to take away from polars at all. It’s damn fast — the benchmarks are hard to beat. I’ve been working on my own library and basically every optimization I can think of is already implemented in polars. I do have a concern with their VC funding/commercialization with cloud. The core library is MIT licensed, but knowing they’ll always have this feauture wall when you want to scale is not ideal. I think it limits the future of the library a lot, and I think long term someone will fill that niche and the users will leave.
- neves 8mo agoIs this the Modern Pandas reference you recommend? https://tomaugspurger.net/posts/modern-1-intro/ https://tomaugspurger.net/posts/modern-1-intro/
- data-ottawa 8mo agoYes it is
- nothrowaways 8mo agoVery well articulated.
- torcete 8mo agoI didn't know about polars, and I can see that they also have a library for R. However, in R, they have a fiercer competition. I wonder how it compares to tidyverse, which is the stablished data analysis library.
- bovermyer 8mo agoAs someone who just encountered Pandas for the first time as part of an Intro to Data Visualization course a few weeks ago, I am now very curious about Polars. The professor doesn't actually care which tool we use as long as we produce nice graphs, so this is as good a time as any to experiment.
- pelasaco 8mo agoare many of the mentioned issues not just some vibe-code sessions away from done?
- noitpmeder 8mo agoGive it a shot and report back when you get them merged
- pelasaco 8mo agonot my circus not my monkeys
- datsci_est_2015 8mo agoMight be cool once PySpark integrates with Polars, but for now like many others I’m stuck with dropping into pandas for non-vectorized operations
- jvican 8mo agoIs there any plan for this?
- datsci_est_2015 8mo agoNot that I’m aware of. The Spark ecosystem seems a little too “stable” to be putting effort into that kind of development. Edit: hah, based on the sibling comment, I stand corrected
- devin-petersohn 8mo agoFunny enough, I actually just (2 weeks ago) added support for streaming from Pyspark to Polars/DuckDB/etc through Arrow PyCapsule. By streaming, I mean actually streaming, not collecting all data at once. It won't be released probably until May/June but it's there: https://github.com/apache/spark/commit/ecf179c3485ba8bac72afd9105892d9798d23f8f https://github.com/apache/spark/commit/ecf179c3485ba8bac72af...
- __mharrison__ 8mo ago"every way" is strong words. Pandas is better for plotting and third party integration.
- vaylian 8mo ago> The design of Pandas is inferior in every way to Polars I used Pandas a lot with Jupyter notebooks. I don't have any experience with Polars. Is it also possible to work with Polars dataframes in Jupyter notebooks?
- disgruntledphd2 8mo agoYes. Most things just work with Polars. The one issue for me is the need for geopandas.
- bikelang 8mo agoAll of this is true and I agree with you - but this comment comes off a bit disrespectful.