6 ms·

Yes, it's annoying negative feature of many tech products. Of course it's natural to want to speak to your target audience (in this case, data scientists who li

Yes, it's annoying negative feature of many tech products. Of course it's natural to want to speak to your target audience (in this case, data scientists who like Pandas but find it annoyingly slow/inflexible), but it's quite alienating to newbies who might otherwise become your most enthusiastic customers. I am the target audience for Polars and have been meaning to try it for several months, but I keep procrastinating about because I feel residual loyalty to Pandas because Wes McKinney (its creator) took the time to write a helpful book about the most common analytical tools: https://wesmckinney.com/book/ https://wesmckinney.com/book/

tomrod 3y ago
Wes has also worked hard to improve a lot of the missteps of pandas, such as through pyarrow, which may prove even more impactful than pandas has been to date. Polars is also a wonderful project!
- adolph 3y ago
  Polars is also based on McKinney’s Arrow project. Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model. https://github.com/pola-rs/polars/blob/main/README.md https://github.com/pola-rs/polars/blob/main/README.md
  - codyvoda 3y ago
    Wes also literally created another Python dataframe project, Ibis, to overcome many of the issues with pandas https://ibis-project.org https://ibis-project.org most data engines and dataframe tools these days use Apache Arrow, it's a bit orthogonal
godelski 3y ago
> it's quite alienating to newbies who might otherwise become your most enthusiastic customers. Newbies are your best target audience too! They aren't already ingrained in a system and have to learn a new framework. They are starting from yours. If a newbie can't get through your docs, you need to improve your docs. But it's strange to me how mature Polars is and that the docs are still this bad. It makes it feel like it isn't in active/continued development. Polars is certainly a great piece of software, but that doesn't mean much if you can't get people to use it. And the better your docs, the quicker you turn noobs into wizards. The quicker you do that, the quicker you offload support onto your newfound wizards.
- theLiminator 3y ago
  > But it's strange to me how mature Polars is and that the docs are still this bad. Interesting, I've personally found them quite good and compared to datafusion or duckdb they're dramatically better. I agree pandas has better docs, but one of the strengths of polars is that I find I often don't need the docs due to putting lots of careful thought into designing a minimal and elegant API, not to mention they're actually care about subtle quirks like making autocomplete, type hinting, etc. work well.
  - godelski 3y ago
    Sounds like we might be coming from different perspectives. I honestly don't use any DF libraries often, and really only Pandas. I used to use pandas a fair amount, but that was years ago, and now I only have to reach for it a few times a year. So maybe the docs are good for people that already have deeper experience. Because I think just the fact that you have used datafusion and duckdb illustrates that you're more skilled in this domain than I am, because I haven't used those haha. But I do think making good docs is quite hard. You usually have multiple audiences that you might not even be aware of. Which makes one of the most important things to do is keep an open ear to listen for them. It's easy to get trapped thinking you got your audience but you're actually closing the door to many more groups (unintentionally). It's also just easy to be focused on the "real" work and not think about docs.
    - sanderjd 3y ago
      What, specifically, is bad about the docs? This whole thread is people who just looked at the home page, saw that it is "DataFrames", but didn't know what that means and came here to complain. Nobody has said anything about issues with the docs for someone who understands what a data frame is (or spent like two minutes looking that up) but is struggling to figure out how to use this library specifically.
      - godelski 3y ago
        I think your experience is probably making it difficult to understand the noob side of things. For me, I've struggled with simply slicing up a dataframe. And as I specified, these aren't tools I use a lot, so the "who understands what a data frame is" probably doesn't apply to me very well and we certainly don't need the pejorative nature suggesting that it is trivially understood or something I should know through divine intervention. I'm sure it's not difficult, but it can take time for things to click. Hell, I can do pretty complex integrals and derivatives and now so much of that seems trivial to me now but I did struggle when learning it. Don't shame people for not already knowing things when they are explicitly trying to learn things. Shame the people that think they know and refuse to learn. There's no reason to not be nice. Having done a lot of teaching I have a note, don't expect noobs to be able to articulate their problems well. They're noobs. They have the capacity to complain but it takes expertise to have clarify that complaint, turning it into a critique. I get that this is frustrating, but being nice turns noobs into experts and often friends too.
lnxg33k1 3y ago
It’s annoying only because it’s on hacker news, because what are the odds of getting on it if you don’t know what is it and don’t have a need for it?
- thecodedmessage 3y ago
  I mean, pretty high. What if your boss just tells you to learn polars, and you don’t know why? Saying what something is, is just good communication, and can help clarify for people who are confused.
  - blowski 3y ago
    These workplaces where bosses tell employees to learn unheard-of tools with zero context sound terrible.
  - azangru 3y ago
    > What if your boss just tells you to learn polars, and you don’t know why? Saying what something is, is just good communication Shouldn't the good communication happen when the boss tells you to learn polars? Like, why are you telling me this, boss; what is it that you need done?
  - lnxg33k1 3y ago
    Guess in the remote event that you're told to learn a new skill that you don't know anything about, you go to pola.rs website and see "DataFrames for the new era" and start getting documentation from there, about what DataFrame is, the website is clearly showing what is it, it's your duty to understand what is it, I would argue that if you knew what DataFrames are you would be saying "Why is it saying something so basic and don't just show me the good stuff?" I for example hate website that try to serve newbies, newbies have a lot of content if they are interested, it's not that all the web needs to serve them
- vaylian 3y ago
  It's annoying because a single leading sentence would be enough to explain a product. Some of the words (for example "Data Frame") in that sentence can be links to other pages if that's necessary. It's a small change but it makes a huge difference.
jeroenjanssens 3y ago
Ritchie Vink (the creator of Polars) deliberately decided not to write a book so that he (and his team) can focus full time on Polars itself. Thijs Nieuwdorp and I are currently working on the O'Reilly book "Python Polars: The Definitive Guide" [1]. It'll be a while before it gets released, but the Early Release version on the O'Reilly platform gets updated regularly. We also post draft chapters on the Polars Discord server [2]. The Discord server is also a great place to ask questions and interact with the Polars team. [1] More information about the book: https://jeroenjanssens.com/pp/ https://jeroenjanssens.com/pp/ [2] Polars Discord server: https://discord.gg/fngBqDry https://discord.gg/fngBqDry
- baq 3y ago
  Slightly offtopic: it's a tragedy that projects like this use discord as the primary discussion forum. It's like slack in that knowledge goes to die there.
  - axus 3y ago
    Shouldn't the knowledge go into a wiki anyways? Reading old discussions in Reddit can turn up interesting things, but it's time consuming and bad information doesn't get fixed later.
  - Fuzzwah 3y ago
    I often see this comment, and every time I think; but having people come to the information AND the community is better for the project.
    - baq 3y ago
      Short term perhaps, but long term having a non-indexed community is inconvenient for newcomers.
      - thijsn 3y ago
        That's why Ritchie is very active on, and often refers to, Stackoverflow as well! Exactly to document frequent questions, instead of losing them to chat history.
davedx 3y ago
I'm a data engineering newbie and I found it very clear, and it gave me an enthusiastic feeling (not an "alienating" feeling). This whole thread just comes across as unmitigated pedantry to me.
- dns_snek 3y ago
  Presumably you were introduced to the concept of DataFrames and how they're used through some other source, because Polars landing page doesn't even bother to mention it's used for data analysis and documentation simply assumes you're already familiar with the core concepts. Compare that to Pandas which starts with the basics, "pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language." It then leads you to "Getting started" guide which features "Intro to pandas" that explains the core concepts.
traceroute66 3y ago
> Yes, it's annoying negative feature of many tech products. Sadly its not only tech products, but also things like security disclosures too. It always follows the same pattern: - Spend $X time coding/researching something. - Spend $not_enough_time documenting it. - Spend $far_too_much_time thinking about / "engaging with the community" in deciding on a cute name, fancy logo and cool looking website.

tomrod 3y ago

Wes has also worked hard to improve a lot of the missteps of pandas, such as through pyarrow, which may prove even more impactful than pandas has been to date. Polars is also a wonderful project!

adolph 3y ago
Polars is also based on McKinney’s Arrow project. Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust using Apache Arrow Columnar Format as the memory model. https://github.com/pola-rs/polars/blob/main/README.md https://github.com/pola-rs/polars/blob/main/README.md
- codyvoda 3y ago
  Wes also literally created another Python dataframe project, Ibis, to overcome many of the issues with pandas https://ibis-project.org https://ibis-project.org most data engines and dataframe tools these days use Apache Arrow, it's a bit orthogonal

godelski 3y ago

> it's quite alienating to newbies who might otherwise become your most enthusiastic customers. Newbies are your best target audience too! They aren't already ingrained in a system and have to learn a new framework. They are starting from yours. If a newbie can't get through your docs, you need to improve your docs. But it's strange to me how mature Polars is and that the docs are still this bad. It makes it feel like it isn't in active/continued development. Polars is certainly a great piece of software, but that doesn't mean much if you can't get people to use it. And the better your docs, the quicker you turn noobs into wizards. The quicker you do that, the quicker you offload support onto your newfound wizards.

theLiminator 3y ago
> But it's strange to me how mature Polars is and that the docs are still this bad. Interesting, I've personally found them quite good and compared to datafusion or duckdb they're dramatically better. I agree pandas has better docs, but one of the strengths of polars is that I find I often don't need the docs due to putting lots of careful thought into designing a minimal and elegant API, not to mention they're actually care about subtle quirks like making autocomplete, type hinting, etc. work well.
- godelski 3y ago
  Sounds like we might be coming from different perspectives. I honestly don't use any DF libraries often, and really only Pandas. I used to use pandas a fair amount, but that was years ago, and now I only have to reach for it a few times a year. So maybe the docs are good for people that already have deeper experience. Because I think just the fact that you have used datafusion and duckdb illustrates that you're more skilled in this domain than I am, because I haven't used those haha. But I do think making good docs is quite hard. You usually have multiple audiences that you might not even be aware of. Which makes one of the most important things to do is keep an open ear to listen for them. It's easy to get trapped thinking you got your audience but you're actually closing the door to many more groups (unintentionally). It's also just easy to be focused on the "real" work and not think about docs.
  - sanderjd 3y ago
    What, specifically, is bad about the docs? This whole thread is people who just looked at the home page, saw that it is "DataFrames", but didn't know what that means and came here to complain. Nobody has said anything about issues with the docs for someone who understands what a data frame is (or spent like two minutes looking that up) but is struggling to figure out how to use this library specifically.
    - godelski 3y ago
      I think your experience is probably making it difficult to understand the noob side of things. For me, I've struggled with simply slicing up a dataframe. And as I specified, these aren't tools I use a lot, so the "who understands what a data frame is" probably doesn't apply to me very well and we certainly don't need the pejorative nature suggesting that it is trivially understood or something I should know through divine intervention. I'm sure it's not difficult, but it can take time for things to click. Hell, I can do pretty complex integrals and derivatives and now so much of that seems trivial to me now but I did struggle when learning it. Don't shame people for not already knowing things when they are explicitly trying to learn things. Shame the people that think they know and refuse to learn. There's no reason to not be nice. Having done a lot of teaching I have a note, don't expect noobs to be able to articulate their problems well. They're noobs. They have the capacity to complain but it takes expertise to have clarify that complaint, turning it into a critique. I get that this is frustrating, but being nice turns noobs into experts and often friends too.

lnxg33k1 3y ago

It’s annoying only because it’s on hacker news, because what are the odds of getting on it if you don’t know what is it and don’t have a need for it?

thecodedmessage 3y ago
I mean, pretty high. What if your boss just tells you to learn polars, and you don’t know why? Saying what something is, is just good communication, and can help clarify for people who are confused.
- blowski 3y ago
  These workplaces where bosses tell employees to learn unheard-of tools with zero context sound terrible.
- azangru 3y ago
  > What if your boss just tells you to learn polars, and you don’t know why? Saying what something is, is just good communication Shouldn't the good communication happen when the boss tells you to learn polars? Like, why are you telling me this, boss; what is it that you need done?
- lnxg33k1 3y ago
  Guess in the remote event that you're told to learn a new skill that you don't know anything about, you go to pola.rs website and see "DataFrames for the new era" and start getting documentation from there, about what DataFrame is, the website is clearly showing what is it, it's your duty to understand what is it, I would argue that if you knew what DataFrames are you would be saying "Why is it saying something so basic and don't just show me the good stuff?" I for example hate website that try to serve newbies, newbies have a lot of content if they are interested, it's not that all the web needs to serve them
vaylian 3y ago
It's annoying because a single leading sentence would be enough to explain a product. Some of the words (for example "Data Frame") in that sentence can be links to other pages if that's necessary. It's a small change but it makes a huge difference.

jeroenjanssens 3y ago

Ritchie Vink (the creator of Polars) deliberately decided not to write a book so that he (and his team) can focus full time on Polars itself. Thijs Nieuwdorp and I are currently working on the O'Reilly book "Python Polars: The Definitive Guide" [1]. It'll be a while before it gets released, but the Early Release version on the O'Reilly platform gets updated regularly. We also post draft chapters on the Polars Discord server [2]. The Discord server is also a great place to ask questions and interact with the Polars team. [1] More information about the book: https://jeroenjanssens.com/pp/ https://jeroenjanssens.com/pp/ [2] Polars Discord server: https://discord.gg/fngBqDry https://discord.gg/fngBqDry

baq 3y ago
Slightly offtopic: it's a tragedy that projects like this use discord as the primary discussion forum. It's like slack in that knowledge goes to die there.
- axus 3y ago
  Shouldn't the knowledge go into a wiki anyways? Reading old discussions in Reddit can turn up interesting things, but it's time consuming and bad information doesn't get fixed later.
- Fuzzwah 3y ago
  I often see this comment, and every time I think; but having people come to the information AND the community is better for the project.
  - baq 3y ago
    Short term perhaps, but long term having a non-indexed community is inconvenient for newcomers.
    - thijsn 3y ago
      That's why Ritchie is very active on, and often refers to, Stackoverflow as well! Exactly to document frequent questions, instead of losing them to chat history.

davedx 3y ago

I'm a data engineering newbie and I found it very clear, and it gave me an enthusiastic feeling (not an "alienating" feeling). This whole thread just comes across as unmitigated pedantry to me.

dns_snek 3y ago
Presumably you were introduced to the concept of DataFrames and how they're used through some other source, because Polars landing page doesn't even bother to mention it's used for data analysis and documentation simply assumes you're already familiar with the core concepts. Compare that to Pandas which starts with the basics, "pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language." It then leads you to "Getting started" guide which features "Intro to pandas" that explains the core concepts.

traceroute66 3y ago

> Yes, it's annoying negative feature of many tech products. Sadly its not only tech products, but also things like security disclosures too. It always follows the same pattern: - Spend $X time coding/researching something. - Spend $not_enough_time documenting it. - Spend $far_too_much_time thinking about / "engaging with the community" in deciding on a cute name, fancy logo and cool looking website.