14 ms·
It's cristal clear that this page has been written for people who already know what they are looking at; the first line of the first paragraph, far from describ
by j1elo 3y ago
It's cristal clear that this page has been written for people who already know what they are looking at; the first line of the first paragraph, far from describing the tool, is about some qualities of it: "Polars is written from the ground up with performance in mind"
And the rest follows the same line.
Anyone could ELI5 what this is and for what needs it is a good solution to use?
EDIT: So an alternative implementation of Pandas DataFrame. Google gave me [0] which explains:
> The pandas DataFrame is a structure that contains two-dimensional data and its corresponding labels. DataFrames are widely used in data science, machine learning, scientific computing, and many other data-intensive fields.
> DataFrames are similar to SQL tables or the spreadsheets that you work with in Excel or Calc. In many cases, DataFrames are faster, easier to use, and more powerful than tables or spreadsheets because they’re an integral part of the Python and NumPy ecosystems.
[0]: https://realpython.com/pandas-dataframe/ https://realpython.com/pandas-dataframe/
- _dain_ 3y agopandas dataframes but faster
- DylanDmitri 3y agoIt’s pandas, but fast. Pandas is the original open source data frame library. Pandas is robust and widely used, but sprawling and apparently slower than this newcomer. The word “data frames” keys in people who have worked with them before.
- elmolino89 3y agoI may be a rare bird starting with R dataframes (still newbie+ level), then python polars (intermediate- ?). Frankly whenever I have to use pandas or df's in R I am not convinced that these are more intuitive/easier to master. I.e. I do not like the concept of row names. Polars can be an overkill for small/medium dataset, but since I have been bitten by corrupted/badly formatted CSVs/TSVs I love the fact that Polars will throw the towel & complain about types/column number mismatches etc. And the fact that it can scale up to millions of rows on a modest workstation compensates the fact that sometimes one can spend hours finding a proper way to manipulate a dataset.
- bee_rider 3y agoAh, like polar bears are a much more aggressive implementation of the idea behind panda bears? That’s a pretty funny name if so.
- ayhanfuat 3y agoNot *original* but probably most commonly used.
- drbaba 3y agoYeah, I believe Pandas was inspired by similar functionality in R.
- froh 3y agoyup I first met data frames in R and pandas is the Python answer to R isn't it
- tomrod 3y agoIf I understand correctly, Pandas original scope was indexed in-memory data frames for use in high frequency trading, making use of the numpy library under the hood. At the time it was written you had JPMC's Athena, GS's platform, and several HFT internal systems (C++ my friends in that space have mentioned). Pandas just is so darn useful! I've been using it since maybe version 0.10, even got to contribute a tiny bit for the sas7bdat handling.
- froh 3y agoindeed it's both: it was created for financial analytics, and it provides R dataframe features to python. thanks for.making me detour into the history of it.
- gmfawcett 3y ago> Pandas is the original open source data frame library ...ehh, not quite. R and its predecessor S have Pandas beat by decades. Pandas wasn't even the first data frame library for Python. But it sure is popular now.
- p4ul 3y agoThat's interesting! I didn't realize there had been prior dataframe libraries in Python! Out of curiosity, what was/were the previous libraries?
- melagonster 3y agoit is built in data structure and function in R.
- p4ul 3y agoOh, yes, I was aware that R (and its predecessor S) have a native dataframe object in the language. It seemed that gmfawcett was indicating that there was a dataframe library in _Python_ that existed prior to Pandas. I was curious what that library was/is, as I'd not heard that before.
- melagonster 3y agook, guess I misunderstood both comments of you two. ´_>`
- gmfawcett 3y agoSorry :) Pandas is undisputed king. But there were multiple bindings from Python into R available in the early 2000's. Some like rpy and rpy2 are still around, others are long defunct. I concede that these weren't standalone dataframe libraries, but rather dataframe features built into a language binding.
- maliker 3y agoPandas has also moved to Apache Arrow as a backend [1], so it’s likely performance will be similar when comparing recent versions. But it’s great to have some friendly competition. [1] https://datapythonista.me/blog/pandas-20-and-the-arrow-revolution-part-i https://datapythonista.me/blog/pandas-20-and-the-arrow-revol...
- thejosh 3y agoMemory and CPU usage is still really high though.
- jasonjmcghee 3y agoNot according to DuckDB benchmarks. Not even close. https://duckdblabs.github.io/db-benchmark/ https://duckdblabs.github.io/db-benchmark/
- keithalewis 3y agoOuch! It is going to take a lot of work to get Polars this fast. If ever.
- hyperpl 3y agoPolars has an OLAP query engine so without any significant pandas overhaul, I highly doubt it will come close to polars in performance for many general case workloads.
- dash2 3y agoThis is a great chance to ELI5: what is an OLAP query engine and why does it make polars fast?
- disgruntledphd2 3y agoPolars can use lazy processing, where it collects all of the operations together and creates a graph of what needs to happen, while pandas executes everything upon calling of the code. Spark tended to do this and it makes complete sense for distributed setups, but apparently is still faster locally.
- dkga 3y agoActually pandas is not the original open source data frame library, perhaps only in Python. There is a very rich tradition in R on data.frames, which includes the unjustly neglected data.table.
- 7thaccount 3y agoYeah. I think Wes McKinney liked the data frames in R, but preferred the programming language of Python. I've heard somewhere that he also got a lot of inspiration from APL.
- Cacti 3y agoR is literally designed to do statistics and has first class support and language feature support for many specialized tasks in statistics and closely related fields. Python is literally designed to be easy to program with in general. Well, it turns out when you’re dealing with terabytes of data and TFLOPS, the programming becomes more important than the math. Not all R devs are happy about this and they are very loud about it. But it shouldn’t really surprise anyone. That is literally how those languages are designed. Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course
- disgruntledphd2 3y ago> Most of the R devs I know like this are just butthurt they are paid less and refuse to switch because they’re obstinate, or they’re a little scared they’re being left behind. first group is all over the place, but the second group tends to skew older of course Look, I started with R and use mostly Python these days, but this is not really a fair take. R is (still) much, much, much better for analytics and graphing (the only decent plotting library in python is a ggplot clone). The big change (and why Python ended up winning) is that integrating R with other tools (like web stuff, for example) is harder than just using Python. pandas (for instance) is like an unholy clone of the worst features from both R and Python. Polars is pretty rocking, though (mostly because it clones from Spark/dplyr/linc). It's another example of Python being the second best language for everything winning out in the marketplace. That being said, if I was starting a data focused company and needed to pick a language, I'd almost certainly build all the DS focused stuff in R as it would be many many times quicker, as long as I didn't need to hire too many people.
- bee_rider 3y agoJust once I’d like to see “this library was written to fulfill head-in-the clouds demands by management that we have some implementation, without regards to quality.”
- minimaxir 3y agoFor posterity, polars was a hobby product that started in 2020: https://news.ycombinator.com/item?id=23768227 https://news.ycombinator.com/item?id=23768227 > As a hobby project I tried to build a DataFrame library in Rust. I got excited about the Apache Arrow project and wondered if this would succeed. > After two months of development it is faster than pandas for groupby's and left and inner joins. I still got some ideas for the join algorithms. Eventually I'd also want to add a query planner for lazy evaluation.
- bee_rider 3y agoDefinitely not intended as a slight toward this project, just (what I thought was) a funny thought about that expression.
- sanderjd 3y agoThat has absolutely no relation to this project. What in the world are you talking about?
- ritchie46 3y agoTrust me. It does. ;)
- drbaba 3y agoAbove that it says “DataFrames for a new era” hidden in their graphics. I believe it’s a competitor to the Python library “Pandas”, which makes it easy to do complex transformations on tabular data in Python.
- CobrastanJorji 3y agoIt seems like it's a disease endemic to data products. Everybody, the big cloud providers and the small data products, build something whose selling point is "I'm the same as Apache X but better." But if you don't know what Apache X is, you have to go read up on that, and its website might say "I'm the same as Whatever Else but better," and you have to go read up on that. I don't want to figure out what a product does by walking a "like X but better" chain and applying diffs in my head. Just tell me what it does! I get that these are general purpose tools with a lot of use cases, but some real quick examples of "this is a good use case" and "this is a bad use case, maybe prefer SQL/nosql/quasisql/hadoop/a CSV file and sed" would be really helpful, please.
- ryandrake 3y agoI run into the same problem. I don't know what Pandas are (besides the bears) and at some point up the "it's like X" chain, I guess you have to stop and admit you're just not the target user of this tech product.
- selcuka 3y ago> I guess you have to stop and admit you're just not the target user of this tech product. On the other hand, how can you become a target user if you don't know that a product category exists?
- esafak 3y agoThat's on you. If you want to become a data engineer and data scientist -- the two software positions most likely to use polars -- get learning. Or don't: learn it when you need it.
- anigbrowl 3y agoYes, it's annoying negative feature of many tech products. Of course it's natural to want to speak to your target audience (in this case, data scientists who like Pandas but find it annoyingly slow/inflexible), but it's quite alienating to newbies who might otherwise become your most enthusiastic customers. I am the target audience for Polars and have been meaning to try it for several months, but I keep procrastinating about because I feel residual loyalty to Pandas because Wes McKinney (its creator) took the time to write a helpful book about the most common analytical tools: https://wesmckinney.com/book/ https://wesmckinney.com/book/
- tomrod 3y agoWes has also worked hard to improve a lot of the missteps of pandas, such as through pyarrow, which may prove even more impactful than pandas has been to date. Polars is also a wonderful project!
- adolph 3y agoPolars is also based on McKinney’s Arrow project. Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model. https://github.com/pola-rs/polars/blob/main/README.md https://github.com/pola-rs/polars/blob/main/README.md
- codyvoda 3y agoWes also literally created another Python dataframe project, Ibis, to overcome many of the issues with pandas https://ibis-project.org https://ibis-project.org most data engines and dataframe tools these days use Apache Arrow, it's a bit orthogonal
- godelski 3y ago> it's quite alienating to newbies who might otherwise become your most enthusiastic customers. Newbies are your best target audience too! They aren't already ingrained in a system and have to learn a new framework. They are starting from yours. If a newbie can't get through your docs, you need to improve your docs. But it's strange to me how mature Polars is and that the docs are still this bad. It makes it feel like it isn't in active/continued development. Polars is certainly a great piece of software, but that doesn't mean much if you can't get people to use it. And the better your docs, the quicker you turn noobs into wizards. The quicker you do that, the quicker you offload support onto your newfound wizards.
- madeofpalk 3y agoI was going to say - it always feels so humbling seeing pages like this. "DataFrames for the new era" okay… maybe I know what data frames are? "Multi-threaded query engine" ahh, so it’s like a database. A graph comparing it to things called pandas, modin, and vaex - I have no clue what any of these are either! I guess this really isn’t for me. It’s a shame because I like to read about new tech or project and try and learn more, even if I don’t understand it completely. But there’s just nothing here for me. This must be what normal people go through when I talk about my lowly web development work…
- theLiminator 3y agoIt's pretty much just an alternative to SQL that's a lot easier/natural to use for more hardcore data analysis. You can much more easily compose the operations you want to run. Just think of it as an API for manipulating tabular data stored somewhere (often parquet files, though they can query many different data sources).
- riku_iki 3y ago> It's pretty much just an alternative to SQL Polars actually has SQL supported, so you can mix and match: e.g. construct DF from SQL.
- youainti 3y agoData tables tend to also be a standard ingestion format for statistical tools in many cases.
- _dain_ 3y agoDataframes and SQL have overlapping functionality, but I wouldn't say that dataframes are an "alternative" to SQL. The tradeoffs are very different. You don't have to worry about minimizing disk reads or think about concurrency issues like transactions or locks, because a dataframe is just an in-memory data structure like a list or a dict, rather than a database. Dataframes also aren't really about relational algebra like SQL is.
- 3y ago
- the__alchemist 3y agoDataframes in Python are a wrapper around 2D numpy arrays, that have labels and various accessors. Operations on them are OOM slower than using the underlying arrays.
- jessekv 3y agoThis is true specifically for Pandas DataFrames. Numpy arrays are themselves just wrappers around c arrays, which are contiguous chunks of memory. Polars supports operations on files larger than memory.
- YetAnotherNick 3y agoI don't know where this myth originated from but I have seen this in multiple places. Even if you think about it 2d numpy arrays can't have different type for different columns.
- theLiminator 3y agoActually numpy has something called a structured array that is pretty much what what you described.
- billyjmc 3y agoWell, if you use structured arrays or record arrays, you can do this (more or less). https://numpy.org/doc/stable/user/basics.rec.html https://numpy.org/doc/stable/user/basics.rec.html
- deleted 3y ago[deleted]
- FrustratedMonky 3y ago"myth originated" It's in the documentation. I just learned pandas recently, and I would have said this same thing. Not because I read through the numpy code, but because I read the documentation. Is it wrong? Can't a user pick up a new tool and trust some documentation without reading through 100's of libraries built on libraries. When was last time someone traced out every dependency, so they can confidently say something is "Myth".
- jmspring 3y agoYou were right that the page is written for those that know what they are looking for, which is just fine. If you are getting started in DS/ML/etc and you have used numpy, pandas, etc. polars is useful in some cases. A simple one, it loads dataframes faster (from experience with a team I help) than pandas. I haven't played enough to know all it's benefits, but yes it's the next logical step if you are in the space using the above mentioned libraries, it's something one will find.
- esafak 3y agoMarketing is a skill that needs to be learned. You have to put yourself in the shoes of a person who knows nothing about your product. This does not come naturally to the engineers who make these products and are used to talking to other specialists like themselves.
- sanderjd 3y agoThis is true in general but I'm not sure it's what's going on here. Marketing is also very concerned with understanding who your target audience(s) are and speaking their language. I think talking about "DataFrames" is exactly that; the target audience of this project knows what that means. What they are interested in is "ok but who cares about data frames? I've been using pandas for like fifteen years", so what you want to tell them is why this is an improvement, how it would help them. Dumbing it down to spend a bunch of space describing what data frames are would just be a distraction. You'd probably lose the target audience before you ever got to the actual benefits of the project.
- aterp 3y agoHad the exact same thought seeing this. Too many of these websites are missing a simple tldr of the thing actually is. Great, it's fast, but fast at what??
- sanderjd 3y agoIt has that simple tldr, it's the very first word, "DataFrames". Everyone in this thread just doesn't know what that means, and that's fine, I get that, but seriously, that's the simple summary. Data frames aren't an obscure or esoteric concept in the data analysis space; quite the opposite.
- sajforbes 3y agoHard agree. People post links to websites with technical descriptions and little basic info all the time, and this is the first time I'm seeing a thread of people complaining about it. If I'm interested in something I see, I start Googling terms; I don't expect a specification for software in a specific field to cater to my beginner-level knowledge.
- Cacti 3y agoI hate this doc style that has become so popular lately. They get so wrapped up in selling you their story that they forget to tell you basic shit. Like what it is. Or how to install it. The PMs literally simplified things so much they simplified the product right ought of the docs.
- orhmeh09 3y agoIt is right there on the page, set to Python by default: > Quick install > Polars is written from the ground up making it easy to install. Select your programming language and get started!
- godelski 3y agoI try to use polars each time I have to do some analysis where dataframes helps. So basically any time I'd reach for pandas, which isn't too often. So each time it's fairly "new". This makes me have a hard time believing everyone that is saying "Pandas but faster" has used Polars, because I can often write Pandas from memory. There's enough subtle and breaking changes that it is a bit frustrating. I really think Polars would be much more popular if the learning curve wasn't so high. It wouldn't be so high if there were just good docs. I'm also confused why there's a split between "User Guide" and "Docs". To all devs: Your docs are incredibly important! They are not an afterthought. And dear god, don't treat them as an afterthought and then tell people opening issues to RTFM. It's totally okay to point people in the right direction without hostility. It even takes less energy! It's okay to have a bad day and apologize later too, you'll even get more respect! Your docs are just as important as your code, even if you don't agree with me, they are to everyone but you. Besides technical debt there is also design debt. If you're getting the same questions over and over, you probably have poor design or you've miscommunicated somewhere. You're not expected to be a pro at everything you do and that's okay, we're all learning. This isn't about polars, but I'm sure I'm not the only one to experience main character devs. It makes me (and presumably others) not want to open issues on __any__ project, not just bad projects, and that's bad for the whole community (including me, because users find mistakes. And we know there's 2 types of software: those with bugs and those that no one uses). Stupid people are just wizards in training and you don't get more wizards without noobs.
- misantroop 3y agoAny new library will have a learning curve. I used Pandas for many years and never could get used to it, never mind memorize most of it. It was always all over the place with no consistency. Switched to Polars a year or so ago and never looked back. I can now usually write it from memory as the method structure is clear. Still running Pandas in a production system that doesn't get many updates but that's all. Even if you like Pandas, you cannot ignore how incredibly slow it is and more importantly, a memory hog.
- killjoywashere 3y agoThe story of Polars seems to be shaping up a bit like the story of Python 3000: everything probably could have been done in a slow series of migrations, but the BDFL was at their limit, and had to start fresh. So it takes 10 years for the community to catch up. In the mean time, there will be a lot of heartache.
- VerminOctopus1 3y agoIt’s not written for you and that’s fine. This is a library targeted at a very specific subset of people and you’re not in it.
- nnevatie 3y agoNoticed exactly the same - there's no description of the library whatsoever on the landing page. It is implied that it is a DataFrame library, whatever that means.
- sanderjd 3y agoMaybe this is sort of like the opposite of how scam emails are purposefully scammy, so that only people who can't recognize scams will fall for them. Only people who know what "a DataFrame library" is - which is an enormous number of people, since this is probably the most broadly known concept in data science / engineering - will keep reading this, and they are the target audience.
- nnevatie 3y ago> which is an enormous number of people While that may be, I think it would make sense to describe the project in a succinct way on the first page a visitor lands.
- nhinck3 3y agoBut you're not the audience? There is very little to gain by tailoring the introduction to people who aren't the audience. You don't go car parts manufacturer expecting an explanation of what an intercooler is.
- sanderjd 3y agoIt is described in a succinct way. "DataFrames" is that description. It's the very first text on the page. It's really the same as having the word "database" be the first text on the landing page of a new database project. If you don't know what the word "database" means, the landing page for a new database project is really not the place to expect to learn about that. The "data frame" concept is not quite as old or broad as the concept of "databases", but it's really not that far off. It's decades old, and is about as close to a universal concept for data work as it's possible to get.
- notatoad 3y agoI think something like dataframes suffers from having a name that isn't obscure enough. You read "dataframes" and think those are two words you know, so you should understand what it is. If they'd called them flurzles you wouldn't feel like you should understand if it's not something you work with.
- WesolyKubeczek 3y agoFor me, “data frames” are forever associated with MPEG
- pama 3y agoIn fairness, the title of the page is “Dataframes for the new Era”. The “Get Started” link below the title links to a document that points to the GitHub page, which explains what the library is about to people with data analysis backgrounds: https://github.com/pola-rs/polars https://github.com/pola-rs/polars
- TylerE 3y agoBut annoyingly, not the <title>, thus the useless HN headline.
- andrewflnr 3y agoI wish HN had secondary taglines we could use to talk about the actual content or relevance of an article apart from its headline.
- vietvu 3y agoIt's fine to me. Tech UI is bad and weird, but not like if you gain 5x customers with better UX.
- Ultimatt 3y agoRight... but the title before the first line reads "DataFrames for the new era". If you don't know what a data frame is then, yes, it's for people who already know that.
- davedx 3y agoI don't use dataframes in my day job but have dabbled in them enough that I found this website pretty easy to digest. You'd really have to be a complete data engineering newbie to not understand it I think? I mean, where do you draw the line? You wouldn't expect a software tool like this to explain what it is in language my grandma would understand, I don't think?
- drbaba 3y ago> You'd really have to be a complete data engineering newbie to not understand it I think? I do occasionally use Pandas in my day job, but I honestly think very few programmers that could have use for a data frame library would describe themselves as a “data engineer” at all. In my case, for example, I’m just a physicist - I don’t work with machine learning, big data, or in the software industry at all. I just use Pandas + Seaborn to process the results of numerical simulations and physical experiments similarly to how someone else might use Excel. Works great.
- mekster 3y agoHow come some submissions don't even describe what it is about than just the name of it? It's really puzzling how everyone is meant to know what it is by its name.
- pixelpoet 3y agoI've mentioned this before and got downvoted because of course everyone is a web dev and knows what xyz random framework (name and version number in the title, nothing else) is.
- Joeboy 3y agoI'm currently getting dragged into "data" stuff, and I get the impression it's a parallel universe, with its own background and culture. A lot of stuff is like "connect to your Antelope or Meringue instances with the usability of Nincompoop and the performance of ARSE2". Anyway, probably the interesting things about polars are that it's like pandas, but uses a more efficient rust "backend" called Arrow (although I think that part's also in pandas now) and something like a "query planner" that makes combining operations more efficient. Typically doing things in polars is much more efficient than pandas, to the extent that things that previously required complicated infrastructure can often be done on a single machine. It's a very friendly competition, created by the main developer of pandas. As far as I can tell everybody loves it and it'll probably supplant pandas over time.
- Galanwe 3y ago> As far as I can tell everybody loves it and it'll probably supplant pandas over time. I've been using pandas heavily, everyday, for something like 8 years now. I also did contribute to it, as well as wrote numpy extensions. That is to say, I'm fairly familiar with the pandas/numpy ecosystem, strengths and weaknesses. Polars is a breeze of fresh air. The API of pandas is a mess: * overuse of polymorphic parameters and return types (functions that accept lists, ndarrays, dataframes, series or scalar) and return differently shaped dataframes or series. * lots of indirections behind layers of trampoline functions that hide default values behind undocumented "=None" default values. * abuse of half baked immutable APIs, favoring "copy everything" style of coding, mixed with half supported, should-have-been-deprecated, in-place variants. * lots and lots of regressions at every new release, of the worst kind ("oh yeah we changed the behavior of function X when there is more than Y NaNs over the window") * Very hard to actually know what is delegated to numpy, what is Cython/pandas, and what is pure python/pandas. * Overall the beast seemed to have won against its masters, and the maintainers seem lost as to what to fix versus what to keep backward compatible. Polars fixes a lot of these issues, but it has some shortcomings as well. Mainly I found that: * the API is definitely more consistent, but also more rigid than pandas. Some things can be very verbose to write. It will take some years for nicer simpler "shortcuts" and patterns to emerge. * The main issue IMHO is polars' handling of cross sectional (axis=1) computations. Polars is _very_ time series (axis=0) oriented, and most cross sectional computations require to transpose the data frame, which is very slow. Pandas has a lot of dedicated axis=1 implementations that avoid a full transposition.