14 ms·
Polars
- hn1986 3y agoThoughts on using Polars in production environments? It's not even 1.0 yet. Where will it be 3 years from now? Note: I do like Polars but have not used it in work setting.
- fifilura 3y agoThere are lots of efforts to make pandas faster. Dask, pandarallel, and this. They all stay at your computer and in your RAM though. I wonder, what is the benefit of using this compared to a truly distributed engine like spark? If your job barely scales maybe that is what you need rather than maximizing your cores or memory usage? (Note, I have not worked with spark, so it is an honest question.)
- bshaver 3y ago[dead]
- benrutter 3y agoDask is actually similar to spark in that it will let you compute outside of your computer over multiple machines. Matt Rocklin (the dask creator) has put some effort into benchmarking across all these recently. You can see the presentation here: https://youtu.be/5YXiRl41DKA?si=5Pt9XQHp5Y1cG7tM https://youtu.be/5YXiRl41DKA?si=5Pt9XQHp5Y1cG7tM If you don't feel like watching a whole presentation, the main take away is local is faster, until it isn't. Polars outperforms spark by a wide margin, until data size gets sufficiently large that things swap over. Some operations can be performance sensitive without having scaling issues. So I guess those kind of jobs are great candidates for something like polars.
- timost 3y agoNice video, thank you. It's interesting to see how each technology behaves depending on the scale of the dataset. Duckdb is definitely killing it.
- fifilura 3y agoAn analogy would be whether I should buy a 2 seated sports car or a family car when I know I have have to drive more than one person around. I think I am a guy who prefer reliability and versatility over speed. (When I was a kid you'd say Alfa Romeo vs Volvo)
- benrutter 3y agoI'm right there with you! For disclosure I'm a minor contributor to dask so probably am a little biased. I guess one side I probably haven't put forward though is that the memory footprint of something like dask/spark is higher because of its overheads. If you don't have scalable resources, then a polars / duckdb option would probably be your most reliable choice (I.e. the one that'll hit the fewest memory errors in the given architecture)
- peter_d_sherman 3y agoRelated: https://www.datarevenue.com/en-blog/pandas-vs-dask-vs-vaex-vs-modin-vs-rapids-vs-ray https://www.datarevenue.com/en-blog/pandas-vs-dask-vs-vaex-v...
- actionfromafar 3y agoHey, what do you peeps use this for? Instead of json.loads() ? :-D
- jerrygenser 3y agoMore like in memory transformations you would do in pandas
- mmaunder 3y agoAnyone got any real-world comparison with Pandas? Like an orders of magnitude wow moment?
- minimaxir 3y agoReal-world performance is complicated since data science covers a lot of use cases. If you're just reading a small CSV to do analysis on it, then there will be no human-perceptible difference between Polars and Pandas. If you're reading a larger CSV with 100k rows, there still won't be much of a perceptible difference. Per this (old) benchmark, there are differences once you get into 10 million rows/500MB+ territory: https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/
- ayhanfuat 3y agoDuckDB is publishing updates to the H20.ai benchmark: https://duckdb.org/2023/11/03/db-benchmark-update.html https://duckdb.org/2023/11/03/db-benchmark-update.html
- alecst 3y agoIn my experience, the biggest difference is in the API. Besides the fact that it's usually faster than pandas, it also feels faster to write and easier to read.
- radus 3y agoIt allowed me to take some code that reads in a bunch of data and performs a few rounds of some pretty standard operations (groupby, filtering, calculating mean/stdevs) and going from pandas to polars allowed me to go from ~1 minute per dataset to 1 second (yes, I tried the Arrow backend for pandas too). This was after spending some time profiling the pandas code and fixing up the slowest parts as best as I could. The translation was pretty straightforward. The output of this pipeline code was a few different dataframes (each to be inserted into a separate table) and each dataframe was output from a function. I was able to migrate one function at a time after asserting that the outputs of the two functions were identical and that all relevant tests passed (I used `to_pandas()` where needed). I'm not sure how much faster I could go, since ~1 second/dataset allowed me to answer some questions that I had that required scanning values for a few parameters. The biggest wins for me were in grouping and merging operations. I'm a complete convert now. The API is simpler and more obvious IMO, and the ability to compose expressions (`polars.Expr`) is awesome. The performance benefits are nice and what motivated me in the first place, but I'm more swayed by the aforementioned benefits.
- appplication 3y agoBeen watching polars for a while. Look promising, would love to test it against spark some day. So much of our infra is on spark though so we’re probably just locked in forever.
- esafak 3y agoSpark feels overwrought, but polars does not scale out yet. https://github.com/pola-rs/polars/issues/5621 https://github.com/pola-rs/polars/issues/5621
- appplication 3y agoInteresting, that is a huge nonstarter for us then. I always thought of polars as “pandas but distributed” but I see I have been incorrect in that assessment.
- theLiminator 3y agoYou might want to consider seeing how far vertical scaling can bring you. https://motherduck.com/blog/big-data-is-dead/ https://motherduck.com/blog/big-data-is-dead/ Is a good read
- appplication 3y agoThat is a good read, and I see most of those points for probably even most midsize companies. I think in this case we are data 1%-ers. We generate and process terabytes of data every day, so the need for horizontal scaling is real.
- sanderjd 3y agoNot super on topic because this is all immature and not integrated with one another yet, but there is a scaled-out rust data-frames-on-arrow implementation called ballista that could maybe? form the backend of a polars scale out approach: https://github.com/apache/arrow-ballista https://github.com/apache/arrow-ballista
- kelseyfrog 3y agoMy data science team evaluated Polars and came back with a mixed bag of results. If there was any performance-critical section, then we would consider employing it, but otherwise it was marginal negative given the overhead of replacing Pandas across dozens of projects.
- gcarvalho 3y agoI think that's the right call. IMHO now is the time to experiment with it, not to replace pandas where it's already working. The API is still seeing some (expected) breaking changes, and could become a maintenance burden across multiple projects. But the API already feels more consistent, and overall seems to be going in the right direction.
- asqueella 3y agoYou made me curious to look up the recent breaking releases https://github.com/pola-rs/polars/releases/tag/py-0.19.0 https://github.com/pola-rs/polars/releases/tag/py-0.19.0 https://github.com/pola-rs/polars/releases/tag/py-0.20.0 https://github.com/pola-rs/polars/releases/tag/py-0.20.0 And their policy about it: https://docs.pola.rs/development/versioning/#deprecation-warnings https://docs.pola.rs/development/versioning/#deprecation-war... Looks like you ought to set aside some time to do updates each quarter, but I do wonder how much breakage can there be in practice, most changes seem pretty niche.
- choppaface 3y agodoes your team use pandasql or mainly the direct pandas api? curious
- theLiminator 3y agoDefinitely don't rewrite all the code, I think it's worth adopting or evaluating in new code though. Especially with the very cheap pandas interop, you can zero-copy to pandas pretty easily if you use the arrow backend.
- esafak 3y agoGoing forward: https://github.com/fugue-project/fugue https://github.com/fugue-project/fugue
- j1elo 3y agoIt's cristal clear that this page has been written for people who already know what they are looking at; the first line of the first paragraph, far from describing the tool, is about some qualities of it: "Polars is written from the ground up with performance in mind" And the rest follows the same line. Anyone could ELI5 what this is and for what needs it is a good solution to use? EDIT: So an alternative implementation of Pandas DataFrame. Google gave me [0] which explains: > The pandas DataFrame is a structure that contains two-dimensional data and its corresponding labels. DataFrames are widely used in data science, machine learning, scientific computing, and many other data-intensive fields. > DataFrames are similar to SQL tables or the spreadsheets that you work with in Excel or Calc. In many cases, DataFrames are faster, easier to use, and more powerful than tables or spreadsheets because they’re an integral part of the Python and NumPy ecosystems. [0]: https://realpython.com/pandas-dataframe/ https://realpython.com/pandas-dataframe/
- _dain_ 3y agopandas dataframes but faster
- DylanDmitri 3y agoIt’s pandas, but fast. Pandas is the original open source data frame library. Pandas is robust and widely used, but sprawling and apparently slower than this newcomer. The word “data frames” keys in people who have worked with them before.
- elmolino89 3y agoI may be a rare bird starting with R dataframes (still newbie+ level), then python polars (intermediate- ?). Frankly whenever I have to use pandas or df's in R I am not convinced that these are more intuitive/easier to master. I.e. I do not like the concept of row names. Polars can be an overkill for small/medium dataset, but since I have been bitten by corrupted/badly formatted CSVs/TSVs I love the fact that Polars will throw the towel & complain about types/column number mismatches etc. And the fact that it can scale up to millions of rows on a modest workstation compensates the fact that sometimes one can spend hours finding a proper way to manipulate a dataset.
- maliker 3y agoBiggest advantage I found when I evaluated it was the API was much more consistent and understandable than the pandas one. Which is probably a given, they’ve learned from watching 20 major versions of pandas get released. However, since it’s much rarer, copilot had trouble writing polars code. So I’m sticking with pandas and copilot for now. Interesting barrier to new libraries in general I hadn’t noticed until I tried this.
- humbleharbinger 3y agoI had a similar experience using danfo.js, another data frame library in js. Copilot straight up hallucinate functionality and method names. Not a big deal because I just read the docs but it was annoying that I couldn't have copilot just spit out what I need.
- im_down_w_otp 3y agoThis is really interesting to see these two posts. I can now imagine where AI tools actually inhibit innovation in many domains simply because they’re optimized for things that are already entrenched and new entrants won’t be in the training data. Further inhibiting adoption compared to existing things and thus further inhibiting enough growth to make it into model updates.
- xpe 3y agoIt is a healthy mindset to see this phenomenon as "interesting". I can get there when I dial up my mindfulness, but my default mode here is rather judgy; as in "please ppl! pick the better tool as evaluated over a 4+ hour timeframe (after you've got some muscle memory for the API) instead of a 15 minute evaluation". Forgive me for ranting here, but have people forgotten how to bootstrap their own knowledge about a new library? Taking notes isn't hard. Making a personal cheat-sheet isn't hard. I say all this AND I use LLMs very frequently to help with technical work. But I'm mindful about the tradeoffs. I will not let the tool steer me down a path that isn't suitable. I'm actually hopeful: there is an unexpected competitive advantage to people who are willing to embrace a little discomfort and take advantage of one's neuroplasticity.
- mmastrac 3y agoWhen we shipped Jupyter support in Deno, `nodejs-polars` was one of the cornerstone library for data science we supported. https://blog.jupyter.org/bringing-modern-javascript-to-the-jupyter-notebook-fc998095081e https://blog.jupyter.org/bringing-modern-javascript-to-the-j... I'm not personally a Data Science guy, but considering how early the JS/Jupyter ecosystem is, it was surprisingly quick to get pola.rs-based analysis up and running in TypeScript. The JS bindings certainly need a bit of love, but hopefully now that it's more accessible we'll see some iteration on it.
- swyx 3y agoTIL. always wanted pandas in JS!
- midnight_shaman 3y agoI am using polars (the rust library) in production. I am mostly satisfied so far. Excellent work!
- p4ul 3y agoSame! And it's been an absolute delight!
- wenc 3y agoI don’t use Polars directly, but instead I use it as a materialization format in my DuckDB workflows. Duckdb.query(sql).pl() is much faster than duckdb.query(sql).df(). It’s zero copy to Polars and happens instantaneously while Pandas takes quite a while if the DataFrame is big. And you can manipulate it like a Pandas DataFrame (albeit with slightly different syntax). It’s greater for working with big datasets.
- petrusnonius 3y agoThanks for this!
- grej 3y agoThis is a really nice insight that I wasn't aware of. Many thanks.
- sanderjd 3y agoVery cool!
- dang 3y agoRelated: Detailed Comparison Between Polars, DuckDB, Pandas, Modin, Ponder, Fugue, Daft - https://news.ycombinator.com/item?id=37087279 https://news.ycombinator.com/item?id=37087279 - Aug 2023 (1 comment) Polars: Company Formation Announcement - https://news.ycombinator.com/item?id=36984611 https://news.ycombinator.com/item?id=36984611 - Aug 2023 (52 comments) Replacing Pandas with Polars - https://news.ycombinator.com/item?id=34452526 https://news.ycombinator.com/item?id=34452526 - Jan 2023 (82 comments) Fast DataFrames for Ruby - https://news.ycombinator.com/item?id=34423221 https://news.ycombinator.com/item?id=34423221 - Jan 2023 (25 comments) Modern Polars: A comparison of the Polars and Pandas dataframe libraries - https://news.ycombinator.com/item?id=34275818 https://news.ycombinator.com/item?id=34275818 - Jan 2023 (62 comments) Rust polars 0.26 is released - https://news.ycombinator.com/item?id=34092566 https://news.ycombinator.com/item?id=34092566 - Dec 2022 (1 comment) Polars: Fast DataFrame library for Rust and Python - https://news.ycombinator.com/item?id=29584698 https://news.ycombinator.com/item?id=29584698 - Dec 2021 (124 comments) Polars: Rust DataFrames Based on Apache Arrow - https://news.ycombinator.com/item?id=23768227 https://news.ycombinator.com/item?id=23768227 - July 2020 (1 comment)
- dangoodmanUT 3y agoso you took my original username
- subroutine 3y ago[flagged]
- mkl 3y agoFrom 2014-03-29, https://news.ycombinator.com/item?id=7494093 https://news.ycombinator.com/item?id=7494093: > A couple of personal points that I may as well insert here. The account I'm now using, dang, was used briefly in 2010 by someone who didn't leave an email address. I feel bad for not being able to ask them if they still want it, so I'm considering this an indefinite loan. If you're the original owner, please email me and I'll give it back to you. The reason I want the name dang is that (a) it's a version of my real name and (b) it's something you say when you make a mistake. You're probably too late.
- Lyngbakr 3y agoLast year, we began testing Polars with TypeScript because I'd heard great things about it for Python in terms of performance and usability. Unfortunately, I ran into bugs that first morning that stopped us in our tracks. I'll definitely give it another shot once they've ironed out the kinks as it looks very promising, but it wasn't ready for prime time for us.
- timenova 3y agoThe Explorer library [0] in Elixir uses Polars underneath it. [0] https://github.com/elixir-explorer/explorer https://github.com/elixir-explorer/explorer
- sanderjd 3y agoWow cool! I had no idea it had made its way into so many libraries across so many languages already! Very impressive for such a new project!
- serjester 3y agoUsed pandas for years and it always felt like rolling a ball uphill - just look at doing something as simple as a join (don't forget to reset the index). Polars feels better than pandas in every way (faster + multi-core, less memory, more intuitive API). The library is still relatively young which has its downsides but in my opinion, at minimum, it deserves to be considered on any new project. Easily being able to leverage the Rust ecosystem is also awesome - I sped up some geospatial code 100X by writing my own plugin to parallelize a function.
- nerdponx 3y ago> just look at doing something as simple as a join (don't forget to reset the index) It's slightly ironic that you mention this, because I always thought the biggest problem with Pandas was its documentation. Case in point: did you know there's a way to join data frames without using the index? It's called "merge" rather than "join". Pandas was originally very heavily inspired by R terminology and usage patterns, where the term "merge" to mean "join" was already commonplace. If I didn't already know R when I started learning Pandas (~2015), I don't think I'd have been able to pick it up quickly at all.
- djhn 3y agoI had to check the R documentation for merge in disbelief, because it didn't ring a bell. Between data.table's [ syntax and dplyr joins I can't remember the last time I've used merge!
- lvncelot 3y agoFor me, Pandas fits in neatly with Matplotlib in the niche category of "R-inspired Python libraries that are somewhat counter-intuitive due to said R-inspiration"
- tda 3y agoMatplotlib is MATLAB inspired, but otherwise your point stands.
- ryanmonroe 3y agoDuckDB maintains a performance benchmark of open source database-like tools, including Polars and Pandas https://duckdblabs.github.io/db-benchmark/ https://duckdblabs.github.io/db-benchmark/
- nerdponx 3y agoI'd like to see Kdb in here.
- rvz 3y agoMeh. Another VC 'open-source' company. > Polars will remain MIT-licensed and the company will sponsor and accelerate the open-source development of Polars. I bet that they will change to a custom business source available license sooner or later. From the about page: [0] > We successfully closed a seed round of approximately 4M$, which was lead by Bain Capital Ventures. Waiting for the day that a seperate closed-source version of polars gets built / forked privately and improvements do not make it back into the source. [0] https://pola.rs/posts/company-announcement/ https://pola.rs/posts/company-announcement/
- Havoc 3y ago> will remain MIT-licensed Seems like fairly strong assurance to me
- ayhanfuat 3y agoYes. And it doesn't really make sense to make the library closed source. The common business model for these kind of tools is to provide consultation, deployment platform, or a distributed version.
- hatmatrix 3y agoDoesn't that imply the opposite? They can take the code base and do anything they want with it.
- mjochim 3y agoSince they are the owners, they can do what they want with it no matter the license. The license only says what others who get a copy of the software can do with it.
- mmaunder 3y agoYeah so this is AI's MovableType moment - maybe. Mojo - a will-be-open-sourced-soon-we-promise fast python alternative targeting AI - is in the same boat. Saying they'll be open source, releasing stuff, but VC backed with the ROI impetus. Quick history lesson: MT owned blogging, got VC backed, changed the license because capitalism, and everyone fled to WordPress which powers over 40% of the Web today. I think looking at the way Matt has run WP and the level of sensitivity he's had to have towards OSS/GPL in order to maintain the ecosystem and community, is instructive when it comes to the viability of an OSS AI VC backed C corp.
- daemonk 3y agoHow well does it work with sklearn/scipy/pytorch?
- ayhanfuat 3y agosklearn is adding support through the dataframe interchange protocol (https://github.com/scikit-learn/scikit-learn/issues/25896 https://github.com/scikit-learn/scikit-learn/issues/25896). scipy, as far as I know, doesn't explicitly support dataframes (it just happens to work when you wrap a Series in `np.array` or `np.asarray`). I don't know about PyTorch but in general you can convert to numpy.
- mmaunder 3y agoFor those familiar with what Pandas and Polars is, I wanted to draw attention to this comment in this thread. Wow! https://news.ycombinator.com/item?id=38920417 https://news.ycombinator.com/item?id=38920417
- wojciem 3y agoHave been using pandas for years and statements about polars definitely seem appealing. Especially around performance, where apply function in pandas (to iterate over rows and derive a new column) can easily be 100x slower than vanilla python. Similar with some pandas APIs culprits like making sure to reset the index after joins and other transformations
- nomilk 3y agoIs the sole appeal of polars (vs, say, pandas) its execution speed? I've found being able to express ideas clearly in code (to aid comprehension now and in the future) to be much more important than shaving off a few seconds of run time. For this reason I think speed alone is not a strong sell point, except specifically in cases where execution times really matter. Analogous somewhat to how ruby/rails might be a 'slow' language/framework (e.g. 600ms when another framework might be 200ms) but multiples faster in facilitating the expression of complex ideas through code, which tends to be the far bigger problem in most software projects.
- singhrac 3y agoThey're similarly expressive, and there are some areas where Polars (taking ideas from years of Pandas development) is ahead, for example with nullable dtypes. That being said, there are many, many times when I would be willing to rewrite code to make it faster or more memory efficient. Just yesterday I rewrote a method from Pandas to Polars to take advantage of that. Time might be cheap (while developing) but memory is expensive. Similarly, if you're writing data science jobs for production, you care about both.
- DanielVZ 3y agoIt strives for a more consistent Dataframe API too. It’s quite subjective but I prefer it
- nerdponx 3y agoThe advantages are: raw performance, an optimizing query engine, streaming/out-of-memory processing support, backed by Arrow so you can load data with zero copy. Some people also prefer the API, which is very similar to that of Spark SQL and might feel more comfortable and consistent to people with professional programming backgrounds.
- sanderjd 3y agoPersonally I prefer its API as well, but that seems to be a more controversial opinion than its often-huge performance wins.
- maronato 3y ago
- didip 3y agoWhat is this thing? Does it aspire to be a Spark replacement?
- esafak 3y agopandas replacement; it runs on a single machine.
- sanderjd 3y agoWhy are there so many low-brow dismissal comments here? Read the docs if you don't know what it is. If you don't know what they mean by "data frames", either move on because this just isn't for you, or do the tiniest possible amount of research into what that means. It's a very common concept that I'm sure every single person who frequents this site is perfectly capable of understanding on their own.
- drivers99 3y ago> paralellism should be parallelism
- theogravity 3y agoIf I didn't just take courses at DataCamp for Python and data science I wouldn't have known this is a replacement for the pandas library.
- __mharrison__ 3y agoI'm about publish Effective Pandas 2 (waiting for pandas 2.2 release) and an getting reviewed for my next book, Effective Polars. Happy to assist and answer any questions I can.
- dr_kiszonka 3y agoBefore I ask my questions, here is an idea for your pandas book, if you haven't covered it already. The support of basic operations, like `round`, depends on the underlying data type (regular float and numpy's float16, float32, float64). Some np floats get rounded while others simply get ignored (they will not get rounded). It took me many hours to figure it out and fix resulting bugs. Maybe others would appreciate some information about this and similar gotchas. Regarding polars, would you have time to answer these questions? 1) How are polars supported by popular data science packages, e.g., for plotting? 2) I know it is a bit silly: is there a way to get around typing `pl.col`, etc. all the time? 3) Besides `tidypolars`, are there any reasonable packages that add support for dplyr-style pipes or operation chaining?
- __mharrison__ 3y ago1 - Polars has support for plotting in the most recent release. Using Matplotlib w/ Polars data tends to work too. Otherwise, you can drop into Pandas to get support where it may be missing. 2 - If you just want the columns, there is no need to make an expression; just pass the string. There are shortcuts like pl.sum(col) instead of pl.col(col).sum(). I guess you could shortcut, c = pl.col, if you really hate typing it... 3 - Polars supports chaining and encourages it out of the box. Anything specific you are looking for?
- rodonn 3y agoFor (2) I like to do: 1. `from polars import col` which at least shortens each use by a few characters 2. For columns that I use very frequently, define a "constant" for them e.g. ID=col("id"). This lets you do df.groupby(ID) instead of df.groupby(pl.col("ID")). Another advantage of defining these column "constants" is that it makes it much easier to refactor to rename all usages of the column (without needing to check whether each string "id" is being used as a column name vs. something else)
- fullofdev 3y agoWhat a domain!
- recursive4 3y agoI recently reached the limits of Pandas running on my 2020 16gb M1. Counting the number of times an element appears in a 1.7B row DataFrame using `df.groupby().size()` would consistently exceed available memory. Rust Polars is able to handle this using Lazy DataFrames / Streaming without issue.
- sweezyjeezy 3y agoFWIW I think df.column.value_counts() is better to use here in pandas.
- recursive4 3y agoIt unfortunately also exceeded available memory. A basic approach which worked was sequentially loading each df from the filesystem, iterating through record hashes, and incrementing a counter; however the runtime was an order of magnitude greater than my final implementation in Polars.
- xwowsersx 3y agoOne comment and a question: firstly, this site is very nice and works well on mobile. Secondly, I wasn't able to find an explanation of how using Polars in Python works. Is it using pyo3 or is this something totally different? Re the performance highlighted in that pie chart, does that hold when using Polars from Python?
- theLiminator 3y agoIt does for the vast majority of things. If anything, due to very aggressive compiler settings, polars in python can be faster. It does use pyo3. Though basically anytime you use a python lambda (the equivalent of pandas apply) on a row basis, it will be limited by python speed/the GIL.
- xwowsersx 3y agoThanks!
- deleted 3y ago[deleted]
- impulser_ 3y agoBuilding high performant, multi-language and multi-platform libraries is probably the best use case for Rust because of the memory safety advantages it has over C and C++. Polars is great example of this. It currently being used in Rust, Python, Javascript, Elixir, R and Ruby.
- LunaSea 3y agoThe amount of memory leaks I got while diving into Polars tell a different story.
- spenczar5 3y agoPolars is cool, but man, I really have come to think that dataframes are disastrous for software. The mess of internal state and confusion of writing functions that take “df” and manipulate it - its all so hard to clean up once you’re deep in the mess. Quivr (https://github.com/spenczar/quivr https://github.com/spenczar/quivr) is an alternative approach that has been working for me. Maybe types are good!
- radus 3y agoTotally agree with the critique, though it bears mentioning that one way that polars differentiates itself from pandas is the expressions API. See [1] for an example. 1: https://kevinheavey.github.io/modern-polars/method_chaining.html#extract-city-names https://kevinheavey.github.io/modern-polars/method_chaining....
- theLiminator 3y agoPolars is a lot better than pandas at maintaining valid state. Because you ideally describe everything in terms of lazy operations, it actually internally keeps track of all your data types at every step until materialization and execution of the query plan. Because of that, you're not going to have the same kind of data type issues you might have in pandas. There are also libraries based off pydantic built for polars dataframes validation (see patito, though it's not mature yet).
- efxhoy 3y agoYes, the dataframe is very tricky to get right function signature wise. I used to write a lot of pandas heavy software and converged on writing most functions to take a dataframe or series and when possible just return a series, then put that series back into the df at the function call site. Handing dfs off to functions that mutate them gets gnarly very quickly.
- perilunar 3y agoNo idea of what this is about, but that pie chart is a crime against data.
- siddboots 3y agoIt resembles a pie chart, but if you think of it instead as like a stop watch, then it makes perfect sense.
- perilunar 3y agoIt would make more sense if each time was an arc instead of a segment. And what does one revolution represent?
- throwaway167 3y agoAmount of time to eat a pie while waiting for your numerical library of choice. Conclusion: Polars gives you indigestion.
- anonu 3y agoWe've been using polars in production for over a year as a replacement to pandas. It's been a good experience: smaller memory footprint, way faster and just more pleasant in general to code. Package is being developed quickly so things get deprecated quickly but I'm not one to complain.
- frogamel 3y agoA few months ago I tried migrating a large pandas codebase to polars. I'm not much of a fan of doing analytics/data pipelining in Python - a complex transformation takes me 2-5x as long in pandas compared to Julia or R (using dataframes.jl & dplyr). Unfortunately polars was not it. Too many bugs on standard operations, unreliable interoperability with pandas (which is an issue since so many libraries require pandas dataframes as inputs), the API is also very verbose for a modern dataframe library, though it's still better than pandas. Hopefully these will get resolved out over time but for now I had the best luck using duckdb on top of pandas, it is as fast as polars but more stable/better interoperability. Eventually I hope the Python dataframe ecosystem gets to the same point as R's, where you have a analytics-oriented dataframe library with an intuitive API (dplyr) that can be easily used alongside a high-performance dataframe library (data.table).
- theLiminator 3y agoMaybe give ibis with the duckdb backend a try, though personally I quite like polars. The devs are pretty fix to respond to issues overall.
- aexl 3y agoNice, you have experience in data frames in R, Python and Julia! Which one of those do you like the most? I know that the ecosystem isn't really comparable, but from your experience, which one is the best to work with for core operations, etc.?
- nerdponx 3y agoNot OP but R data.table + dplyr is an unbeatable combo for data processing. I handily worked with 1bn record time series data on a 2015 MBP. The rest of the tidyverse stuff is OK (like forcats), but the overall ecosystem is a little weird. The focus on "tidy" data itself is nice up to a point, but sometimes you want to just move data around in imperative style without trying to figure out which "tidy verb" to use, or trying to learn yet another symbol interpolation / macro / nonstandard eval system, because they seem to have a new one every time I look. Pandas is a real workhorse overall. Data.table is like a fast sports car with a very complicated engine, and Pandas is like a work van. It's a little of everything and not particularly excellent at anything and that's ok. Also its index/multiindex system is unique and powerful. But data.table always smoked it for single-process in-memory performance. Until DuckDB and Polars, there was no Python equivalent of data.table at all. They're great when you want high performance, native Arrow (read: Parquet) support, and/or an interface that feels more like a programming library than a data processing tool. If you're coming from a programming background, or if you need to do some data processing or analytics inside of production system, those might be good choices. The Polars API will also feel very familiar to users of Spark SQL. For geospatial data, Pandas is by far superior to all options due to GeoPandas and now SpatialPandas. There is an alpha-stage GeoPolars library but I have no idea who's working on it or how productive they will be. If you had to learn one and only one, Pandas might still be the best option. Python is a much better general-purpose language than R, as much as I love R. And Pandas is probably the most flexible option. Its index system is idiosyncratic among its peers, but it's quite powerful once you get used to using it, and it enables some interesting performance optimization opportunities that help it scale up to data sets it otherwise wouldn't be able to handle. Pandas also has pretty good support for time series data, e.g. aggregating on monthly intervals. Pandas also has the most extensibility/customizability, with support for things like custom array back ends and custom data types. And its plotting methods can help make Matplotlib less verbose. I've never gotten past "hello world" with Julia, not for lack of interest, but mostly for lack of time and need. I would be interested to hear about that comparison as well.
- smcleod 3y agoWhy is everyone's marketing team underlining a random word in their slogans these days? Is the most important takeaway not to be forgotten from this product the word "era"?
- imgabe 3y agoThere must be a corollary to Greenspun's Tenth Rule (https://en.wikipedia.org/wiki/Greenspun's_tenth_rule) https://en.wikipedia.org/wiki/Greenspun's_tenth_rule) that any sufficiently complicated data analysis library contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of SQL. I use Pandas from time to time and I'll probably try this out, but I always find myself wishing I'd just started with shoving whatever data I'm working with into Postgres. It's not like I'm some database expert either, I'm far more comfortable with Python, but the facilities for selecting, sorting, filtering, joining, etc tabular data are just way better in SQL.
- sanderjd 3y agoYou could do that, but it would likely both perform significantly worse (if you're doing "analytical" kinds of queries) and be a lot less flexible and expressive. But you may want to look into DuckDB, which has a sql implementation that is not ad hoc, bug ridden, slow, or incomplete (though I honestly don't know about the formality of its specification). And it is compatible with polars :)
- snthpy 3y agoI recommend you look at DuckDB and the duckdb-prql extension. DuckDB allows you to work on your Polars and Pandas (and any data on Arrow format) directly using SQL without needing any data copying or duplication. The duckdb-prql allows you to use PRQL (prql-lang.org) which gives you all the power and universality of SQL with the ergonomics and DX of Polars or Pandas (IMHO). Disclaimer: I'm a PRQL contributor.
- petrusnonius 3y agoThank you for suggesting DuckDB. Looks brilliant!
- theLiminator 3y agoImo lazy dataframe syntax is a far superior frontend to a query engine. Polars also has SQL support, but really the frontend isn't generally where bugs come from but instead come from the query engine. Postgres would be an order of magnitude slower than OLAP query engines for the types of queries that people do with them.
- joshsyn 3y ago[dead]
- hresvelgr 3y agoI used this at work about a year ago to build a statistics platform and it was damn good, chewing through gigabytes of data quickly and with no hassle. Having never worked with such a library before I found myself asking my colleague who'd worked with Pandas a lot of questions which he was able to answer easily due to the overlap. My only critique of the Rust crate is that it's not as well documented as the Python API and the Rust API required a lot more unwrapping and error handling than other Rust crates which was quite tedious.
- hansvm 3y agoCrazy coincidence, on a whim I learned a bit of polars this morning to prep for a data science interview. All I really wanted was "SQL but cleanly FFI mixable with python," and the API flowed nicely. Every time I said a certain function or syntax should exist, the dir() or help() info confirmed it existed. It was fantastic. Pandas was a nightmare trying to remember how to slice and dice everything, and this was a breath of fresh air.
- gigatexal 3y agoI like the api for this much better than pandas but ymmv
- ekianjo 3y agoand here is a Polars library if you use R: https://github.com/pola-rs/r-polars https://github.com/pola-rs/r-polars
- _Wintermute 3y agoThe R syntax is not kind on the eyes. I know that's a shallow dismissal, but I think it would really start to irritate me if I had to read that all day.
- rodonn 3y agoIf you prefer dplyr syntax you can use https://tidypolars.etiennebacher.com/ https://tidypolars.etiennebacher.com/
- jh_zab 3y agoI have ported a few internal libraries to polars from pandas and had great results. I never liked pandas much due to its annoying API, indices and single thread only implementation (we usually get a 10x performance boost at least and for me that also means improvements in productivity). Also, pandas never handled NULLs properly. This should now work with the pyarrow backend in pandas, but we can’t read our parquet files generated by PySpark. With polars it mostly just works, but we use pyarrow to read/write parquet. Overall I can recommend it, conversion from/to pandas DataFrames was never an issue as well.
- jamesblonde 3y agoThere is a new open source columnar lakehouse gaining traction with low cost data processing and storage, with Polars as part of it: 1. Querying and Data processing in Polars or DuckDB 2. Metadata (for transactions, time-travel) table formats (Iceberg, Hudi, Delta) 3. Storage with Parquet on S3. With Polars/DuckDB, you can process up to 100+ GB of data at a time on a single VM, turn it off when you're not using it. It's a lower overhead, lower time-to-value stack than Spark. And even if you have TBs of data, so long as you only process ~100GB at a time, it works fine.
- tsss 3y agoWhy the 100GB hard limit? If you can stream from disk you should be able to process infinite data sets if the operations allow it. If you need to do stuff like deduplication, then it will depend on how much RAM you have available.
- jamesblonde 3y agoThere's no "limit", just a kind of practical size I have seen. It's a single VM. You will need NVMe to prevent disk being a bottleneck (NVMe is now 8 GB/s vs 200 MB/s for S3).
- DEDLINE 3y agoKnow of any papers / articles on this stack?
- ImageXav 3y agoLike with many such projects, it's very helpful if you use DataFrames in isolation, but it lacks support from the wider scientific ecosystem. I've found that using polar will often break other common data scientific packages such as scikit-learn. This unfortunately often makes it impractical in the wild.
- benrutter 3y agoIts worth drawing attention to the awesome work pandas are doing building in support for arrow data types right now. This actually means you can have zero copy swaps between polars, pandas and duckdb with very little runtime impact. Definitely a big help if your using a non-pandas df library since the conversion to pandas when needed is now really cheap.
- theLiminator 3y agoJust convert it to pandas/numpy at the edges?
- swaraj 3y agoAlways happy to see new stuff on the block, but hard to leave pandas and python ecosystem for this Not sure where this fits in to any workflow tbh, with sufficiently large datasets, you will inevitably need spark (which has same API as pandas)
- F-W-M 3y agoIs there a good book/overview on how dataframes and OLAP query engines work under the hood?
- sidcool 3y agoApart from being in memory, any other advantages of data frames over a Postgres table with indexes?
- qwertox 3y agoAny recommendations on which binary format to use when I want to store all the DataFrames to disk in order to load them at a later point? My data as an indented JSON file takes up around 800 MB.
- congoe 3y agoParquet works well as it natively support Apache Arrow, the underlying data structure.
- benrutter 3y agoI'm really excited about Polars and it's speed performance is super impressive buuutt. . . It annoys me to see vaex, modin and dask all compared on the same benchmarks. For anyone who doesn't use those libraries, they are all targeted towards out-of-core data processing (i.e. computing across multiple machines because your data is too big). Comparing them to a single core data frame library is just silly, and they will obviously be slower because they necessarily come with a lot of overhead. It just wouldn't make sense to use polars in the same context as those libraries, so seeing them presented in benchmarks as if they are equivalents is a little silly. And on top of that, duckdb, which you might use in the same context as polars and is faster than polars in a lot of contexts, isn't included in the benchmarks. The software engineering behind polars is amazing work and there's no need to have misleading benchmarks like this.
- codyvoda 3y agoIbis, a Python dataframe created by the creator of pandas, uses DuckDB as the default backend and generally beats Polars on these benchmarks (with exceptions on some queries)
- oreilles 3y agoI don't know about the others but you can use Dask on a single machine, and it's also the easiest way to use Dask. It allows parallelizing operations by splitting dataframes into partitions that get processed in individual cores on your machine. Performance boost over pandas can be 2x with zero config, and I've seen up to 5x on certain operations.
- munro 3y agoI have been using Polars on and off for the past few years, but now I've been using it 99% exclusively for the past 2–3 months. I would say it's ready for prime time! I don't get any @TODO seg faults anymore lol I don't know if I notice the speed difference over Pandas most of the time, but I do find the way of expressing transformations way more intuitive than Pandas, it's more similar to SQL.
- elbear 3y agoRomanians chuckling at the domain name
- softwaredoug 3y agoAre there resources for converting a Pandas extension array (ie custom data type) to Polars? I have a column type that's a searchable full-text index, I'd like to have it support both if possible.
- brap 3y agoNewbie question: When would you use dataframes (e.g Pandas, Polars...) and when would you use tensors (Pytorch, TF...)? Are the usecases completely distinct or is there overlap?
- codyvoda 3y agofairly distinct, there is of course some overlap. you could technically do (mostly?) everything with tensors that you could with dataframes, but generally dataframes are for analyzing and transforming data for ETL/analytic workloads. tensors are how machine learning models understand data, i.e. before training a neural network (or LLM) at some point text is converted to numbers in tensors you still transform data in tensors, but generally that's one-hot encoding or transposing or other transformation done right before model training. before that, you might use a dataframe to cleanup strings, aggregate timeseries data, etc. hope that makes sense. so yes there's some overlap, but generally they're distinct toolsets that would be used together for an end-to-end ML project
- ryukoposting 3y agoI rarely use Pandas (just haven't come across it much in my work) but I almost want to come up with a side project just so I can fiddle with this. It's making some big promises. By the way, that pie chart on the home page is a crime.
- kavalg 3y agoI am curious how does this compare to nvidia's rapids.ai / cudf.
- Bjornlandberg 3y ago[dead]