8 ms·

Replacing Pandas with Polars

103 points by bjourne 4y ago

aorist 4y ago
The ergonomics of grouping and aggregation in R are really much better because libraries can make of its non-standard evaluation[^0] (which in other cases also makes the language a nightmare to deal with). Compare: pd_df.groupby(['date'])['failure'].count() # pandas pl_df.groupby(pl.col('date')).agg(pl.count('failure')) # polars dt[, .N, date] # R data.table In both Pandas and Polars, the specification of the date has to be a string inside a list or method call, but in R it can be a bare token. [^0]: http://adv-r.had.co.nz/Computing-on-the-language.html http://adv-r.had.co.nz/Computing-on-the-language.html
- barumrho 4y ago
  While it's shorter, it seems more magical? How does it know to count `failure`?
  - deleted 4y ago
    [deleted]
  - aorist 4y ago
    It doesn't count `failure` — just the number of rows. But neither does the pandas version: `pd_df.groupby(['date'])['failure'].count()` and `pd_df.groupby(['date']).count()` are the same except the former returns a single `pd.Series` with the count and the latter produces a `pd.DataFrame` where each column has the same count (not super useful). e.g. > iris.groupby('species').count() sepal_length sepal_width petal_length petal_width species setosa 50 50 50 50 versicolor 50 50 50 50 virginica 50 50 50 50 vs. > iris.groupby('species')['sepal_length'].count() species setosa 50 versicolor 50 virginica 50
    - iamlemec 4y ago
      I believe `count` will only give you the number of non-null rows, so the numbers from the first command could differ by column if there were null values. You can also use the `size` command to get the total number of rows, and that will return a `pd.Series` with or without a column specifier.
- wokwokwok 4y ago
  Having migrated 1000s of lines of legacy r, all I can say is… yes, but then you have you to use r. (: R is not a replacement for pandas. R is it’s own special little painful ecosystem, loved by people who don’t have to maintain the code they write. You can complain all you like about pandas, but at the end of the day, it’s python. Python tooling works with it. The python ecosystem works with it. It’s not without faults, but at least you’ll have a community of people to help when things go wrong. R, not so much. (Spoken as jaded developer who had to support r on databricks, which is deep in the hell of “well, it’s not really a tier one language” even from their support team)
  - jasonpbecker 4y ago
    Having written tens of thousands of lines of R code that I've been maintaining and using for production pipelines for 9 years... Sounds like you worked with (or wrote) really bad R code.
    - wokwokwok 4y ago
      The point I’m making isn’t that you can write bad r code; you can write bad code in any language. …the point I’m making is that when you already have bad r code, it’s a massive pain in the ass. Bad python is terrible too, but lots of people know python, and it’s easy to find help to unduck bad python code and turn it into maintainable code. That has not been my experience with r. Ever. At any organisation. Your experience may vary. (:
      - nequo 4y ago
        > when you already have bad r code, it’s a massive pain in the ass. Do you think it’s because R code tends to be written by statisticians and stats-adjacent domain experts who don’t necessarily know how to write clean code while Python code has at least some input from actual programmers? Or is this really down to the language itself?
      - bovinejoni 4y ago
        Can you elaborate please? This hasn’t been my experience whatsoever, curious what the issues have been
- minimaxir 4y ago
  data.table syntax is indeed more concise but harder to parse later, which is more important in a collaborative environment. dplyr syntax IMO is the best balance between clarity and nonredundancy as evident in pandas/polar code.
- tryptophan 4y ago
  R is easy and fun to use. R is also impossible to understand whta is actually going on.
  - miohtama 4y ago
    R is Perl for math
- mutant_self 4y ago
  I would do the Pandas and Polars examples differently: ``` pd_df['date'].value_counts() # pandas pl_df.select(pl.col('date').value_counts()) # polars ``` Note: we could also do it the Pandas way in Polars but square bracket indexing in Polars is not recommended
Lyngbakr 4y ago
> You know, Pandas feels like Airflow, everyone keeps talking about its demise, but there it is everywhere … used by everyone. Wait — did I miss the memo? What's replacing Airflow?
- massaman_yams 4y ago
  Prefect, dagster, or maybe one could argue google cloud composer is different enough from vanilla airflow that it earns a spot on this list? But you're right, Airflow is still the de facto standard, there's just increasing awareness of pain points with it.
ericpauley 4y ago
Given these examples, I don't really see why anyone would replace Pandas with Polars for existing workloads. Memory's not an issue because they already bought the DIMMs (which have gotten insanely cheap). Personally I don't find the speed of Pandas to be that bad if you use it smartly (e.g., not parsing data more often than you have to).
- jerpint 4y ago
  I agree nothing in that article was a compelling argument
- minimaxir 4y ago
  Working with Actual Big Data (tm) will inevitably hit issues with both aspects in pandas, where polars can theoretically help. Granted, the real correct solution for manipulating Actual Big Data (tm) is at the data warehouse level itself before it even hits Python but that's a topic for another day.
  - __mharrison__ 4y ago
    Ponder is working on making Pandas actually work with really big data. (Disclaimer: I'm an advisor for them.)
- __mharrison__ 4y ago
  This article suffers from the same thing that most Pandas articles or books suffer from. Showing operations in isolation. I rarely do one thing to a dataset. Show me what real world code looks like. Pandas and Polars are both awesome. Pandas is more entrenched and is a solid choice these days (and getting better for big data). Polars is an awesome up and coming library. Competition is good for the Python Data ecosystem. (Disclaimer: I've authored multiple Pandas books and have taught courses on both Pandas and Polars.)
extr 4y ago
I tried to seriously use Polars in a side project and while I do like that the API surface is a little smaller, and cleaner, it's quite verbose. And I ran into a few bugs doing "basic" things that made me think it's not quite ready for prime time yet. I think with a bit more development and a more interactive-friendly API it would be awesome though.
- ericpauley 4y ago
  I've noticed a trend lately (though I doubt it's new) of systems/frameworks/whatever coming out promising to unseat the existing offering by being simpler, more opinionated, etc. But it turns out all that complexity was because the underlying problem is actually complex, so as the new simpler solution gains adoption it feature creeps its way back to being the same as the old. I feel like I'm witnessing the start of that here.
  - extr 4y ago
    I agree in the general case but I think here the problem is more that to really make these types of packages to air-tight production grade status, you need a critical mass of users to basically field test it and find all the edge cases and bugs. This is what I see polars struggling with right now. The issue I ran into was with the drop_nan function. A pretty common task. I was using it in a weird way, but if polars wants to succeed with it's small-but-feature-complete API surface, it needs it's verbs to be fully, reliably composable. No bugs allowed. The big thing pandas has going for it is that it's already been through this field testing. All the bugs have been ironed out by the hundreds of thousands of users. And the compute/memory efficiency is "good enough" for most use-cases, that those who need better efficiency can spend some time to optimize it. Sure the API is confusing and bad, but most people just kind of muddle through using stackoverflow answers from 2017 and it works well enough. I've stanned for data.table (the R package) on HN before and I'll do it again now. It's a great example of a package that has been thoroughly field tested as well as having a simple, elegant API that allows you to do basically anything. So we know it's possible!
    - lr1970 4y ago
deleted 4y ago
[deleted]
anyfactor 4y ago
I have tried polars for a couple of week and I am one of those weird guys who likes pandas syntax more than sql. Honestly, this is one of those "I will just wait until it (Pandas) gets better" thing. Anyone who uses Pandas and SQL extensively knows that whatever question you might have with them someone already has an answer for you. One the other hand Polars is new and I feel like the Polars community is pushing the better syntax to wrangle data just doesn't feel right. I am not smart enough to put three different wrangling/query syntax in my brain. I am hopeful about duckdb mostly because how friendly the people behind the project is. But honestly they really need to improve their csv reader operation. The data type recognition for auto read csv needs work. And duckdb people knows about CSV read is the reason polars has an edge on them and I know they are working on it. But at the end of all that, I will just wait for Pandas to get faster and better.
- smohare 4y ago
  This sounds like it’s spoken from someone who doesn’t understand a lick about sql. Pandas has one of the worst APIs I’ve ever seen. Truly a blight on the data processing landscape.
  - brahbrah 4y ago
    If you have a dataframe of power plant capacities and a dataframe of power plant capacity reductions how would you figure out the available capacities of all your power plants. In pandas it would be `capacities_df - reductions_df`. How would you do it in polars or sql, it’s not nearly as nice. Pandas has the benefit of allowing you to work with data in a relational or ndarray style. Rather than one or the other. The api does incur some bloat due to that, but that ability is very valuable.
- RandomWorker 4y ago
  Here here, I agree. In the most recent versions we have seen speed increases. The fact that polars exist shows that there is a tone of low hanging fruit. There is also dask to increase parallelism and performance which I’ve used on some massive datasets 200GB+
VyseofArcadia 4y ago
Surely I'm not the only one who saw the headline and thought it would be about bears. I thought we were going to get some interesting zoo logistics or something.
- shadowofneptune 4y ago
  Yeah, the literal meaning sounds absolutely disastrous and I'd love/dread to see it.
sixdimensional 4y ago
Or replace Pandas with Pandas API on Spark [1]? [1] https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/best_practices.html https://spark.apache.org/docs/latest/api/python/user_guide/p...
fumeux_fume 4y ago
Yeah, Pandas is so awful they copied the api almost exactly, but with less control over functionality.
chazeon 4y ago
The benchmarks[1] from polars' websites seems really promising, and I am going to try it on my next jupyter notebook. But what I wonder really is about parsing time, this is not shown in bench marks. I my previous project, I use pandas' read_table to avoid for loop to parse molecular dynamics trajectories LAMMPS. The trajectories are pure text file with size of a few Gigs. Compared to a pure Python implementation ~3-5 min, pandas reduce it to 20 secs. I wonder how Polars works. To be honest, compared to parsing, data manipulation such as the benchmark showcased, is just not so costly. [1]: https://www.pola.rs/benchmarks.html https://www.pola.rs/benchmarks.html
- maegul 4y ago
  My understanding is that polars can get fast(er) parsing mainly/only by leveraging arrow and parquet files.
  - Jweb_Guru 4y ago
    As far as I know, it's much faster than Pandas (and other competitors) at parsing plain CSVs as well.
  - ritchie46 4y ago
    Author of polars here. That's incorrect. I have put a lot of effort in polars' csv parser for instance and it is one of the fastest csv parsers out there. This has nothing to do with leveraging arrow. We control all our parsers and read directly into arrow memory. This differs from pandas, which utilizes pyarrow for reading parquet and then finally has to copy the arrow memory over to pandas memory.
    - maegul 4y ago
      Apologies! I got that from some blog post somewhere I believe, not from any personal wackereren our judgement (which I should have signalled better in my post). Nonetheless, would reading a parquet file with polars be faster than reading a csv? Also thanks for polars! Great contribution data science!
ibgeek 4y ago
I wish the article addressed integration with seaborn and other libraries. I’m sure Polars can do most of what I want in isolation, but I’m not sure how well it integrates with everything else.
- ZeroCool2u 4y ago
  I use it with plotly and it's great.
- maegul 4y ago
  This is the weird thing that's happening right now, and maybe it's not a good thing. Any pipelines that have been built from the daraframe onward (datavis libraries being the big ones) have been based on pandas (or rather its DataFrame API) as the de facto standard. Additionally, other libraries or wrappers have sought to replicate or be compatible with the pandas API. But, the one universal complaint about pandas was always its API! Moving on from it, I'd say, is a substantial part of the motivation behind something like polars. So we're in some unpleasant "worse is better" territory here, and to me, it feels like python really missed the mark by too much. The pandas API is "fine", but really, we could do with better. One option is to just treat the pandas DataFrame as the translation layer. If you need to dump tabular data from your library into a datavis library, then `plot(my_df.to_pandas())`. Hopefully that conversion overhead never becomes too much of a headache. If pandas gets full Arrow compatibility (last I checked, they were behind R and tidyverse on this??), then it shouldn't really be an issue anymore and Wes Mckinney can hopefully stop getting complaints about the pandas API!.
bayesian_horse 4y ago
And in ten years we'll replace Polars with the new library "Koalas".
- pieix 4y ago
  Ten years is a lot of time, but is it enough time for the current Koalas package [0] to truly die and leave the name for the next dataframe library? [0] https://koalas.readthedocs.io/en/latest/ https://koalas.readthedocs.io/en/latest/
davnn 4y ago
It‘s a bit sad that Pandas has become the default API for data manipulation in Python. I think it‘s less intuitive than any of the other API I‘ve worked with, for example R‘s tidyverse, Julia‘s DataTable and even Mathematica‘s approach makes more sense than Pandas in my opinion.
- happymellon 4y ago
  I found that my problem with Pandas, was that is was quite un-pythonic. Almost everything required reading to grasp and creation of several dummy tasks, nothing was intuitive and in the end while some of the functions helped I am not entirely sure that the overhead was worth it compared to researching alternative solutions.
- snthpy 4y ago
  [Disclaimer: PRQL core dev here] I'm hoping that PRQL [0] will one day become the universal API for tabular/relational data. We're targeting SQL as the backend in the first iteration given it's universality but there are early plans to support other backends. You can already use PRQL with Pandas, the tidyverse, shell and pretty much any database. See my presentation [1]. PRQL reads very similarly to dplyr, and in my (biased) opinion, actually a bit better than dplyr because it can do away with some of the punctuation due to being its own language. For questions see our Discord [2] and if you would like to see PRQL in more places, file an issue on Github [3]. Some examples below: ## Pandas ```python #!pip install pyprql import pandas as pd import pyprql.pandas_accessor df = pd.read_csv("data/customers.csv") df.prql.query('filter country=="Germany"') ``` ## tidyverse ```sh mkdir -p ~/.local/R_libs R -q -e 'install.packages("prqlr", repos = "https://eitsupi.r-universe.dev", lib="~/.local/R_libs/")' ``` ```R library(prqlr, lib.loc="~/.local/R_libs/") library("tidyquery") " from mtcars filter cyl > 6 sort [-mpg] select [cyl, mpg] " |> prql_to_sql() |> query() ``` ### PRQL ```prql from employees filter start_date > @2021-01-01 derive [ gross_salary = salary + (tax ?? 0), gross_cost = gross_salary + benefits_cost, ] filter gross_cost > 0 group [title, country] ( aggregate [ average gross_salary, sum_gross_cost = sum gross_cost, ] ) filter sum_gross_cost > 100_000 derive id = f"{title}_{country}" derive country_code = s"LEFT(country, 2)" sort [sum_gross_cost, -country] take 1..20 ``` [0]: https://prql-lang.org/ https://prql-lang.org/ [1]: https://github.com/snth/normconf2022/blob/main/notebooks/normconf.ipynb https://github.com/snth/normconf2022/blob/main/notebooks/nor... [2]: https://discord.gg/N63hWUhw https://discord.gg/N63hWUhw [3]: https://github.com/prql/prql https://github.com/prql/prql
IceDane 4y ago
Pandas is just the worst. I mean, it's obviously powerful, but it's still the worst. It suffers from the same problem as so many other python libraries which were clearly designed by scientists rather than software engineers. The design philosophy seems to be that having 100s of ways to do every single thing is "convenient". In reality, it makes almost every single function in the pandas library waaaaaaay too complicated. Nearly all of them have different "modes", and sometimes choosing one thing over another will absolutely cripple the performance of your application. I hadn't heard about polars, but I'm currently using pandas in a part of our stack, and since it's not a lot of code yet, I might just be able to replace pandas. Thanks!
- deleted 4y ago
  [deleted]
- d4rkp4ttern 4y ago
  Yesterday I spent over 45 mins dealing with this error: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas https://stackoverflow.com/questions/20625582/how-to-deal-wit... “SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.”
  - brahbrah 4y ago
    So in pandas you might get that warning when you’re trying to do an in place operation like so: df.loc['2023-01-01': '2023-12-31'] *= 2 This doubles every row in 2023 in your frame. In polars there’s no mutating operations, everything is pure, so you won’t see any warnings of that nature, but what you will see instead is having to do something like this to solve the same problem: df.with_column([ pl.when(pl.col('date').is_between( datetime(2023, 1, 1), datetime(2023, 12, 31), )).then(pl.col(x) * 2) .otherwise(pl.col(x)) .alias(x) for x in df.columns if x != 'date' ]) So there’s definitely some trade offs, but mutating operations are sometimes worth learning how to use them properly to avoid that pandas warning (or know when to ignore it, because unfortunately there can be false positives).
__tmk__ 4y ago
Why is the article using PNG images for the code samples? I guess because the author thinks it looks prettier this way, but I don't really see it.
- deleted 4y ago
  [deleted]
captainmuon 4y ago
I like that there is an alternative to Pandas, but unfortunately it uses the same dataframes paradigm. I know I'm in the minority, but I never liked dataframes. They are too "R"-like, don't fit into Python, and try to do too many things at once. Columns have names, except when they don't. Things (rows? individual cells?) have types which is sometimes enforced and sometimes not. If you don't follow a recipe exactly, your dataframe will be subtly different and might not work. A lot of code mixes mutation in place and creating new dataframes, and so on. And Pandas lends itself to a special chained method syntax which is almost it's own DSL in Python.
- ritchie46 4y ago
  > Columns have names, except when they don't. Things (rows? individual cells?) They are not a fit for every problem. Please don't try to use them for that. For normalized tabular data, it is a good fit. > have types which is sometimes enforced and sometimes not. If you don't follow a recipe exactly, your dataframe will be subtly different and might not work. Polars is really strict on types. A type is always determined by the input datatype and the operation applied. You can check the schema before you run the query. Polars will produce the same output datatype as defined by its logical plan. > A lot of code mixes mutation in place and creating new dataframes, and so on All polars methods are pure, e.g. they don't mutate the original.
  - jiggunjer 4y ago
    You mean if your memory can only hold a single large dataframe, polars won't work well?
    - ayembee 4y ago
      No; there is an OOC (out-of-core, eg: > RAM) streaming mode. https://github.com/pola-rs/polars#handles-larger-than-ram-data https://github.com/pola-rs/polars#handles-larger-than-ram-da...
- pydry 4y ago
  The problem isnt really with the paradigm but that it is untyped. There are like 10 3rd party solutions for this but ffs it should be built in.
rubenfiszel 4y ago
One thing that high-performance polars/arrow enables is to build ETL without data sharding because in most cases, all your data fits into one node. Jupyter and local scripts become sufficient and you avoid a lot of complexity. We hesitated a lot before mentioning ETL in the OSS project windmill [1] but since we support any python including polars, passing data between steps through shared folders and since polars is blazingly fast, it is a suprisingly good system for ETL waiting for the need of using spark/materialize directly. [1]: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill
- RockyMcNuts 4y ago
  Vertical is the new horizontal, because of stuff like Epycs with 96 cores/192 threads, AWS instances with 24TB of RAM, M2 MacBook Pros with 96GB RAM, 12 cores, ludicrous SOC bandwidth. What used to be 'big data' fits on an instance, you only need shards/clusters for 'ludicrous data'.
  - RockyMcNuts 4y ago
    Also, it would make me so happy if someone did a proper benchmark of - pandas - polars - modin with various backends - DuckDB and Postgres SQL from Jupyter/python with the usual SQL dataframe magic - R - on medium and 'pretty big data' i.e. 10GB dataframes with millions of rows. - maybe on consumer Intel, M2, and Epyc with lots of cores
rgavuliak 4y ago
A blog like this only scratches the surface. A couple of simple examples don't really help decide on whether it's a good substitute.
d4rkp4ttern 4y ago
Fugue is an interesting library in this space , though I haven’t tried it https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark, Dask and Ray without any rewrites.
liminal 4y ago
I've been centering Arrow in my data workflows, which has led to a system that uses Ray for distributed computation, Polars for single node data frame computation on Arrow data sets, and Ray Data for distributed data frames using Arrow. I love the speed and Polars' expressive API, and I love that I can use any other Arrow tool on the data without any serialization overhead. That said, using Ray places limits on that, since it can distribute data across machines, breaking the Arrow model of in-memory data sharing between tools. I know that Ray Data is starting to use Polars internally, but I wish I could use the Polars API over distributed Ray Data sets so I could write code once and not worry about whether Ray has split it across machines.

103 points by bjourne 4y ago

The ergonomics of grouping and aggregation in R are really much better because libraries can make of its non-standard evaluation[^0] (which in other cases also makes the language a nightmare to deal with). Compare: pd_df.groupby(['date'])['failure'].count() # pandas pl_df.groupby(pl.col('date')).agg(pl.count('failure')) # polars dt[, .N, date] # R data.table In both Pandas and Polars, the specification of the date has to be a string inside a list or method call, but in R it can be a bare token. [^0]: http://adv-r.had.co.nz/Computing-on-the-language.html http://adv-r.had.co.nz/Computing-on-the-language.html

barumrho 4y ago
While it's shorter, it seems more magical? How does it know to count `failure`?
- deleted 4y ago
  [deleted]
- aorist 4y ago
  It doesn't count `failure` — just the number of rows. But neither does the pandas version: `pd_df.groupby(['date'])['failure'].count()` and `pd_df.groupby(['date']).count()` are the same except the former returns a single `pd.Series` with the count and the latter produces a `pd.DataFrame` where each column has the same count (not super useful). e.g. > iris.groupby('species').count() sepal_length sepal_width petal_length petal_width species setosa 50 50 50 50 versicolor 50 50 50 50 virginica 50 50 50 50 vs. > iris.groupby('species')['sepal_length'].count() species setosa 50 versicolor 50 virginica 50
  - iamlemec 4y ago
    I believe `count` will only give you the number of non-null rows, so the numbers from the first command could differ by column if there were null values. You can also use the `size` command to get the total number of rows, and that will return a `pd.Series` with or without a column specifier.
wokwokwok 4y ago
Having migrated 1000s of lines of legacy r, all I can say is… yes, but then you have you to use r. (: R is not a replacement for pandas. R is it’s own special little painful ecosystem, loved by people who don’t have to maintain the code they write. You can complain all you like about pandas, but at the end of the day, it’s python. Python tooling works with it. The python ecosystem works with it. It’s not without faults, but at least you’ll have a community of people to help when things go wrong. R, not so much. (Spoken as jaded developer who had to support r on databricks, which is deep in the hell of “well, it’s not really a tier one language” even from their support team)
- jasonpbecker 4y ago
  Having written tens of thousands of lines of R code that I've been maintaining and using for production pipelines for 9 years... Sounds like you worked with (or wrote) really bad R code.
  - wokwokwok 4y ago
    The point I’m making isn’t that you can write bad r code; you can write bad code in any language. …the point I’m making is that when you already have bad r code, it’s a massive pain in the ass. Bad python is terrible too, but lots of people know python, and it’s easy to find help to unduck bad python code and turn it into maintainable code. That has not been my experience with r. Ever. At any organisation. Your experience may vary. (:
    - nequo 4y ago
      > when you already have bad r code, it’s a massive pain in the ass. Do you think it’s because R code tends to be written by statisticians and stats-adjacent domain experts who don’t necessarily know how to write clean code while Python code has at least some input from actual programmers? Or is this really down to the language itself?
    - bovinejoni 4y ago
      Can you elaborate please? This hasn’t been my experience whatsoever, curious what the issues have been
minimaxir 4y ago
data.table syntax is indeed more concise but harder to parse later, which is more important in a collaborative environment. dplyr syntax IMO is the best balance between clarity and nonredundancy as evident in pandas/polar code.
tryptophan 4y ago
R is easy and fun to use. R is also impossible to understand whta is actually going on.
- miohtama 4y ago
  R is Perl for math
mutant_self 4y ago
I would do the Pandas and Polars examples differently: ``` pd_df['date'].value_counts() # pandas pl_df.select(pl.col('date').value_counts()) # polars ``` Note: we could also do it the Pandas way in Polars but square bracket indexing in Polars is not recommended

Lyngbakr 4y ago

> You know, Pandas feels like Airflow, everyone keeps talking about its demise, but there it is everywhere … used by everyone. Wait — did I miss the memo? What's replacing Airflow?

massaman_yams 4y ago
Prefect, dagster, or maybe one could argue google cloud composer is different enough from vanilla airflow that it earns a spot on this list? But you're right, Airflow is still the de facto standard, there's just increasing awareness of pain points with it.

ericpauley 4y ago

Given these examples, I don't really see why anyone would replace Pandas with Polars for existing workloads. Memory's not an issue because they already bought the DIMMs (which have gotten insanely cheap). Personally I don't find the speed of Pandas to be that bad if you use it smartly (e.g., not parsing data more often than you have to).

jerpint 4y ago
I agree nothing in that article was a compelling argument
minimaxir 4y ago
Working with Actual Big Data (tm) will inevitably hit issues with both aspects in pandas, where polars can theoretically help. Granted, the real correct solution for manipulating Actual Big Data (tm) is at the data warehouse level itself before it even hits Python but that's a topic for another day.
- __mharrison__ 4y ago
  Ponder is working on making Pandas actually work with really big data. (Disclaimer: I'm an advisor for them.)
__mharrison__ 4y ago
This article suffers from the same thing that most Pandas articles or books suffer from. Showing operations in isolation. I rarely do one thing to a dataset. Show me what real world code looks like. Pandas and Polars are both awesome. Pandas is more entrenched and is a solid choice these days (and getting better for big data). Polars is an awesome up and coming library. Competition is good for the Python Data ecosystem. (Disclaimer: I've authored multiple Pandas books and have taught courses on both Pandas and Polars.)

extr 4y ago

I tried to seriously use Polars in a side project and while I do like that the API surface is a little smaller, and cleaner, it's quite verbose. And I ran into a few bugs doing "basic" things that made me think it's not quite ready for prime time yet. I think with a bit more development and a more interactive-friendly API it would be awesome though.

ericpauley 4y ago
I've noticed a trend lately (though I doubt it's new) of systems/frameworks/whatever coming out promising to unseat the existing offering by being simpler, more opinionated, etc. But it turns out all that complexity was because the underlying problem is actually complex, so as the new simpler solution gains adoption it feature creeps its way back to being the same as the old. I feel like I'm witnessing the start of that here.
- extr 4y ago
  I agree in the general case but I think here the problem is more that to really make these types of packages to air-tight production grade status, you need a critical mass of users to basically field test it and find all the edge cases and bugs. This is what I see polars struggling with right now. The issue I ran into was with the drop_nan function. A pretty common task. I was using it in a weird way, but if polars wants to succeed with it's small-but-feature-complete API surface, it needs it's verbs to be fully, reliably composable. No bugs allowed. The big thing pandas has going for it is that it's already been through this field testing. All the bugs have been ironed out by the hundreds of thousands of users. And the compute/memory efficiency is "good enough" for most use-cases, that those who need better efficiency can spend some time to optimize it. Sure the API is confusing and bad, but most people just kind of muddle through using stackoverflow answers from 2017 and it works well enough. I've stanned for data.table (the R package) on HN before and I'll do it again now. It's a great example of a package that has been thoroughly field tested as well as having a simple, elegant API that allows you to do basically anything. So we know it's possible!
  - lr1970 4y ago

deleted 4y ago

[deleted]

anyfactor 4y ago

I have tried polars for a couple of week and I am one of those weird guys who likes pandas syntax more than sql. Honestly, this is one of those "I will just wait until it (Pandas) gets better" thing. Anyone who uses Pandas and SQL extensively knows that whatever question you might have with them someone already has an answer for you. One the other hand Polars is new and I feel like the Polars community is pushing the better syntax to wrangle data just doesn't feel right. I am not smart enough to put three different wrangling/query syntax in my brain. I am hopeful about duckdb mostly because how friendly the people behind the project is. But honestly they really need to improve their csv reader operation. The data type recognition for auto read csv needs work. And duckdb people knows about CSV read is the reason polars has an edge on them and I know they are working on it. But at the end of all that, I will just wait for Pandas to get faster and better.

smohare 4y ago
This sounds like it’s spoken from someone who doesn’t understand a lick about sql. Pandas has one of the worst APIs I’ve ever seen. Truly a blight on the data processing landscape.
- brahbrah 4y ago
  If you have a dataframe of power plant capacities and a dataframe of power plant capacity reductions how would you figure out the available capacities of all your power plants. In pandas it would be `capacities_df - reductions_df`. How would you do it in polars or sql, it’s not nearly as nice. Pandas has the benefit of allowing you to work with data in a relational or ndarray style. Rather than one or the other. The api does incur some bloat due to that, but that ability is very valuable.
RandomWorker 4y ago
Here here, I agree. In the most recent versions we have seen speed increases. The fact that polars exist shows that there is a tone of low hanging fruit. There is also dask to increase parallelism and performance which I’ve used on some massive datasets 200GB+

VyseofArcadia 4y ago

Surely I'm not the only one who saw the headline and thought it would be about bears. I thought we were going to get some interesting zoo logistics or something.

shadowofneptune 4y ago
Yeah, the literal meaning sounds absolutely disastrous and I'd love/dread to see it.

sixdimensional 4y ago

Or replace Pandas with Pandas API on Spark [1]? [1] https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/best_practices.html https://spark.apache.org/docs/latest/api/python/user_guide/p...

fumeux_fume 4y ago

Yeah, Pandas is so awful they copied the api almost exactly, but with less control over functionality.

chazeon 4y ago

The benchmarks[1] from polars' websites seems really promising, and I am going to try it on my next jupyter notebook. But what I wonder really is about parsing time, this is not shown in bench marks. I my previous project, I use pandas' read_table to avoid for loop to parse molecular dynamics trajectories LAMMPS. The trajectories are pure text file with size of a few Gigs. Compared to a pure Python implementation ~3-5 min, pandas reduce it to 20 secs. I wonder how Polars works. To be honest, compared to parsing, data manipulation such as the benchmark showcased, is just not so costly. [1]: https://www.pola.rs/benchmarks.html https://www.pola.rs/benchmarks.html

maegul 4y ago
My understanding is that polars can get fast(er) parsing mainly/only by leveraging arrow and parquet files.
- Jweb_Guru 4y ago
  As far as I know, it's much faster than Pandas (and other competitors) at parsing plain CSVs as well.
- ritchie46 4y ago
  Author of polars here. That's incorrect. I have put a lot of effort in polars' csv parser for instance and it is one of the fastest csv parsers out there. This has nothing to do with leveraging arrow. We control all our parsers and read directly into arrow memory. This differs from pandas, which utilizes pyarrow for reading parquet and then finally has to copy the arrow memory over to pandas memory.
  - maegul 4y ago
    Apologies! I got that from some blog post somewhere I believe, not from any personal wackereren our judgement (which I should have signalled better in my post). Nonetheless, would reading a parquet file with polars be faster than reading a csv? Also thanks for polars! Great contribution data science!

ibgeek 4y ago

I wish the article addressed integration with seaborn and other libraries. I’m sure Polars can do most of what I want in isolation, but I’m not sure how well it integrates with everything else.

ZeroCool2u 4y ago
I use it with plotly and it's great.
maegul 4y ago
This is the weird thing that's happening right now, and maybe it's not a good thing. Any pipelines that have been built from the daraframe onward (datavis libraries being the big ones) have been based on pandas (or rather its DataFrame API) as the de facto standard. Additionally, other libraries or wrappers have sought to replicate or be compatible with the pandas API. But, the one universal complaint about pandas was always its API! Moving on from it, I'd say, is a substantial part of the motivation behind something like polars. So we're in some unpleasant "worse is better" territory here, and to me, it feels like python really missed the mark by too much. The pandas API is "fine", but really, we could do with better. One option is to just treat the pandas DataFrame as the translation layer. If you need to dump tabular data from your library into a datavis library, then `plot(my_df.to_pandas())`. Hopefully that conversion overhead never becomes too much of a headache. If pandas gets full Arrow compatibility (last I checked, they were behind R and tidyverse on this??), then it shouldn't really be an issue anymore and Wes Mckinney can hopefully stop getting complaints about the pandas API!.

bayesian_horse 4y ago

And in ten years we'll replace Polars with the new library "Koalas".

pieix 4y ago
Ten years is a lot of time, but is it enough time for the current Koalas package [0] to truly die and leave the name for the next dataframe library? [0] https://koalas.readthedocs.io/en/latest/ https://koalas.readthedocs.io/en/latest/

davnn 4y ago

It‘s a bit sad that Pandas has become the default API for data manipulation in Python. I think it‘s less intuitive than any of the other API I‘ve worked with, for example R‘s tidyverse, Julia‘s DataTable and even Mathematica‘s approach makes more sense than Pandas in my opinion.

happymellon 4y ago
I found that my problem with Pandas, was that is was quite un-pythonic. Almost everything required reading to grasp and creation of several dummy tasks, nothing was intuitive and in the end while some of the functions helped I am not entirely sure that the overhead was worth it compared to researching alternative solutions.
snthpy 4y ago
[Disclaimer: PRQL core dev here] I'm hoping that PRQL [0] will one day become the universal API for tabular/relational data. We're targeting SQL as the backend in the first iteration given it's universality but there are early plans to support other backends. You can already use PRQL with Pandas, the tidyverse, shell and pretty much any database. See my presentation [1]. PRQL reads very similarly to dplyr, and in my (biased) opinion, actually a bit better than dplyr because it can do away with some of the punctuation due to being its own language. For questions see our Discord [2] and if you would like to see PRQL in more places, file an issue on Github [3]. Some examples below: ## Pandas ```python #!pip install pyprql import pandas as pd import pyprql.pandas_accessor df = pd.read_csv("data/customers.csv") df.prql.query('filter country=="Germany"') ``` ## tidyverse ```sh mkdir -p ~/.local/R_libs R -q -e 'install.packages("prqlr", repos = "https://eitsupi.r-universe.dev", lib="~/.local/R_libs/")' ``` ```R library(prqlr, lib.loc="~/.local/R_libs/") library("tidyquery") " from mtcars filter cyl > 6 sort [-mpg] select [cyl, mpg] " |> prql_to_sql() |> query() ``` ### PRQL ```prql from employees filter start_date > @2021-01-01 derive [ gross_salary = salary + (tax ?? 0), gross_cost = gross_salary + benefits_cost, ] filter gross_cost > 0 group [title, country] ( aggregate [ average gross_salary, sum_gross_cost = sum gross_cost, ] ) filter sum_gross_cost > 100_000 derive id = f"{title}_{country}" derive country_code = s"LEFT(country, 2)" sort [sum_gross_cost, -country] take 1..20 ``` [0]: https://prql-lang.org/ https://prql-lang.org/ [1]: https://github.com/snth/normconf2022/blob/main/notebooks/normconf.ipynb https://github.com/snth/normconf2022/blob/main/notebooks/nor... [2]: https://discord.gg/N63hWUhw https://discord.gg/N63hWUhw [3]: https://github.com/prql/prql https://github.com/prql/prql

IceDane 4y ago

Pandas is just the worst. I mean, it's obviously powerful, but it's still the worst. It suffers from the same problem as so many other python libraries which were clearly designed by scientists rather than software engineers. The design philosophy seems to be that having 100s of ways to do every single thing is "convenient". In reality, it makes almost every single function in the pandas library waaaaaaay too complicated. Nearly all of them have different "modes", and sometimes choosing one thing over another will absolutely cripple the performance of your application. I hadn't heard about polars, but I'm currently using pandas in a part of our stack, and since it's not a lot of code yet, I might just be able to replace pandas. Thanks!

deleted 4y ago
[deleted]
d4rkp4ttern 4y ago
Yesterday I spent over 45 mins dealing with this error: https://stackoverflow.com/questions/20625582/how-to-deal-with-settingwithcopywarning-in-pandas https://stackoverflow.com/questions/20625582/how-to-deal-wit... “SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.”
- brahbrah 4y ago
  So in pandas you might get that warning when you’re trying to do an in place operation like so: df.loc['2023-01-01': '2023-12-31'] *= 2 This doubles every row in 2023 in your frame. In polars there’s no mutating operations, everything is pure, so you won’t see any warnings of that nature, but what you will see instead is having to do something like this to solve the same problem: df.with_column([ pl.when(pl.col('date').is_between( datetime(2023, 1, 1), datetime(2023, 12, 31), )).then(pl.col(x) * 2) .otherwise(pl.col(x)) .alias(x) for x in df.columns if x != 'date' ]) So there’s definitely some trade offs, but mutating operations are sometimes worth learning how to use them properly to avoid that pandas warning (or know when to ignore it, because unfortunately there can be false positives).

__tmk__ 4y ago

Why is the article using PNG images for the code samples? I guess because the author thinks it looks prettier this way, but I don't really see it.

deleted 4y ago
[deleted]

captainmuon 4y ago

I like that there is an alternative to Pandas, but unfortunately it uses the same dataframes paradigm. I know I'm in the minority, but I never liked dataframes. They are too "R"-like, don't fit into Python, and try to do too many things at once. Columns have names, except when they don't. Things (rows? individual cells?) have types which is sometimes enforced and sometimes not. If you don't follow a recipe exactly, your dataframe will be subtly different and might not work. A lot of code mixes mutation in place and creating new dataframes, and so on. And Pandas lends itself to a special chained method syntax which is almost it's own DSL in Python.

ritchie46 4y ago
> Columns have names, except when they don't. Things (rows? individual cells?) They are not a fit for every problem. Please don't try to use them for that. For normalized tabular data, it is a good fit. > have types which is sometimes enforced and sometimes not. If you don't follow a recipe exactly, your dataframe will be subtly different and might not work. Polars is really strict on types. A type is always determined by the input datatype and the operation applied. You can check the schema before you run the query. Polars will produce the same output datatype as defined by its logical plan. > A lot of code mixes mutation in place and creating new dataframes, and so on All polars methods are pure, e.g. they don't mutate the original.
- jiggunjer 4y ago
  You mean if your memory can only hold a single large dataframe, polars won't work well?
  - ayembee 4y ago
    No; there is an OOC (out-of-core, eg: > RAM) streaming mode. https://github.com/pola-rs/polars#handles-larger-than-ram-data https://github.com/pola-rs/polars#handles-larger-than-ram-da...
pydry 4y ago
The problem isnt really with the paradigm but that it is untyped. There are like 10 3rd party solutions for this but ffs it should be built in.

rubenfiszel 4y ago

One thing that high-performance polars/arrow enables is to build ETL without data sharding because in most cases, all your data fits into one node. Jupyter and local scripts become sufficient and you avoid a lot of complexity. We hesitated a lot before mentioning ETL in the OSS project windmill [1] but since we support any python including polars, passing data between steps through shared folders and since polars is blazingly fast, it is a suprisingly good system for ETL waiting for the need of using spark/materialize directly. [1]: https://github.com/windmill-labs/windmill https://github.com/windmill-labs/windmill

RockyMcNuts 4y ago
Vertical is the new horizontal, because of stuff like Epycs with 96 cores/192 threads, AWS instances with 24TB of RAM, M2 MacBook Pros with 96GB RAM, 12 cores, ludicrous SOC bandwidth. What used to be 'big data' fits on an instance, you only need shards/clusters for 'ludicrous data'.
- RockyMcNuts 4y ago
  Also, it would make me so happy if someone did a proper benchmark of - pandas - polars - modin with various backends - DuckDB and Postgres SQL from Jupyter/python with the usual SQL dataframe magic - R - on medium and 'pretty big data' i.e. 10GB dataframes with millions of rows. - maybe on consumer Intel, M2, and Epyc with lots of cores

rgavuliak 4y ago

A blog like this only scratches the surface. A couple of simple examples don't really help decide on whether it's a good substitute.

d4rkp4ttern 4y ago

Fugue is an interesting library in this space , though I haven’t tried it https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark, Dask and Ray without any rewrites.

liminal 4y ago

I've been centering Arrow in my data workflows, which has led to a system that uses Ray for distributed computation, Polars for single node data frame computation on Arrow data sets, and Ray Data for distributed data frames using Arrow. I love the speed and Polars' expressive API, and I love that I can use any other Arrow tool on the data without any serialization overhead. That said, using Ray places limits on that, since it can distribute data across machines, breaking the Arrow model of in-memory data sharing between tools. I know that Ray Data is starting to use Polars internally, but I wish I could use the Polars API over distributed Ray Data sets so I could write code once and not worry about whether Ray has split it across machines.