6 ms·

"the API makes Pandas look good" It sparks joy in my heart whenever I see shade cast against pandas.

by chaps 4y ago

avaldez_ 4y ago
What's wrong with pandas? Honest question. I'm a bit new to ML. Also, what's the alternative?
lordgroff 4y ago
Every time I open up pandas I jealously remember the expressive beauty of R for these tasks. But because we're all "serious" of course we must use Python for production lest we not be serious.
- laichzeit0 4y ago
  To be fair, taking R to production is a goddamn nightmare.
  - chaxor 4y ago
    R is a trash of a language. It doesn't have any sense of coherency to it at all. They keep trying to fix the underlying problems by ducktaping paradigms on to it over and over (S3, S4, R6, etc). There's never a clear sense of the best way to do anything, but plenty of options to do a thing in a very hacky 'script-kiddy' way. Looking out at the community of different projects it becomes clear that everyone is pretty lost as to what design principles should be used for certain tasks, so every repo has its own way of doing things (I know personal style occurs in other languages, but commonalities are much less recognizable in R projects). It's tragic that such a large community uses it.
    - lordgroff 4y ago
      Yep, this is a mark of someone that's never used R but has heard a lot of incredibly ill informed criticism around it. One look of dplyr code over pandas would of course disabuse anyone of the notion that R is trash and the tragedy is Python will in the current state never have anything like that. That's the advantage of the language being influenced by Lisp vs not.
      - tomrod 4y ago
        I've heavily used R several times. I agree that it is a trash language and that, outside that many frontier academic ideas are available and some plotting preferences are solidly prescriptive, it should be thrown into the trash bin. Python, Julia when it gets its druthers for TTFP, Octave, Fortran, C, and eventually Rust. These are the tools I've found in use over and over and over again across business, government, and non-profits. Everywhere R is used by the org I have seen major gaps in capacity to deliver specifically because R doesn't scale well.
        nerdponx 4y ago
        Try to separate the language from its standard library. Neither one is "trash". I agree that the standard library is what you might call "a chaotic disorganized mess".
        tomrod 4y ago
        I'm not emotionally invested in tools so am happy to identify the user experience and operational experience as "trash." "Trash", despite its connotations of lacking value, is really just a chaotic disorganized mess of something made by artifice with dubious reclaim/reuse/recycle value. Being a subjective assessment, it is natural that one person's trash is a treasure to another.
        nerdponx 4y ago
        I take issue with your implication that I'm emotionally invested in something when I shouldn't be. You are free to dislike R and not use it, but to claim that it's "trash" is to wrongly disavow its usefulness for the many people that do find it useful, and to cast aspersions on the judgement of all those people.
        tomrod 4y ago
        Hey, I apologize here, my point on emotional investment was that I, personally, am not emotionally invested in it and did not mean to cast aspersions at you for your defense of the language nor at people who have preferences for it. Specifically I meant that I'm comfortable enough in my understanding of the language to classify it and it's standard library as better in the garbage bin relative to alternatives available. It's fine that people like it. What's good about it isn't unique, and what's unique about it isn't that great. And there are certainly switching costs for some orgs to consider.
    - jmt_ 4y ago
      Trash language is a bit harsh. I'm not sure I would try to put an R project into production or build a huge project with it but, at the very least, R/R Studio was the best scientific calculator I've ever used. Was particularly great during college
    - jjgreen 4y ago
      It's not trash, it's functional, look: x <- 3
whatever1 4y ago
I have never seen a worse documented library. Initially I thought that they were lazy, now I realize that it cannot be documented because it is a total mess of a library held together with tape. Close second is the plotly library.
- __mharrison__ 4y ago
  Genuinely curious what you have against the Pandas documentation. It has some of the best docstrings I've seen. (I also wrote a Pandas book or two... So there's that)
  - chaps 4y ago
    Docstrings are one thing, but functionality discovery, picking up from scratch, troubleshooting, etc are... not fun, nor easy with the documentation. If you know it well already and use it a lot it's easier to forgive its documentation faults since you can waive off the problems as "that's just learning something new". But for a lot of people who use it infrequently its documentation is a frustrating mess. Simple problems turn into significant time sinks of trying to find which page of the documentation to look at. A lot of issues are made worse by shit-awful interop between libraries that claim to fully support dayaframes, but often fail in non-obvious ways... meaning back to the documentation mines. I'd argue that because there's a market for a single author to write two books about it is indicative of documentation problems.
    - __mharrison__ 4y ago
      Fair enough. I'm highly biased and my recent book is the most popular Pandas book currently, so it is evidence that folks prefer opinionated documentation. However, I always though the 10 minutes to Pandas page was decent for getting started. I picked up Polars recently and thought it was more difficult than Pandas because there wasn't any quick intro docs. What projects have great introductory docs for you? Also, I am curious to learn more about the specifics of interop libraries you are referring to. Learning a new tool is generally a challenge. I think another challenge with a lot of data tools is that non-programmers tend to be the major audience. I make my living teaching "non-programmers" how to use these tools. That said, I always teach "go to the docstrings and stay in your environment (to not break flow) if you can." The pydata docstrings are better than most, including Python (the language).
dekhn 4y ago
What bothers me the most is the egregious data types for any argument. If it's a string, do this. If it's a list, do that. If it's a dictionary of lists, do this other thing. No, I want you to force me to provide my data in the right way and raise a noisy exception if I don't.
- nerdponx 4y ago
  Series and DataFrame have "alternate constructors" for this purpose, and the loc/iloc accessors give you a bit more control. I agree that the magic type auto-detection is a bit too magical and sloppy, but you have to realize that data analysts and scientists have historically been incredibly sloppy programmers who wanted as much magic as possible. It's only in recent years that researchers have begun to value some amount of discipline in their research code.

"the API makes Pandas look good" It sparks joy in my heart whenever I see shade cast against pandas.

by chaps 4y ago

"the API makes Pandas look good" It sparks joy in my heart whenever I see shade cast against pandas.

avaldez_ 4y ago

What's wrong with pandas? Honest question. I'm a bit new to ML. Also, what's the alternative?

lordgroff 4y ago

Every time I open up pandas I jealously remember the expressive beauty of R for these tasks. But because we're all "serious" of course we must use Python for production lest we not be serious.

laichzeit0 4y ago
To be fair, taking R to production is a goddamn nightmare.
- chaxor 4y ago
  R is a trash of a language. It doesn't have any sense of coherency to it at all. They keep trying to fix the underlying problems by ducktaping paradigms on to it over and over (S3, S4, R6, etc). There's never a clear sense of the best way to do anything, but plenty of options to do a thing in a very hacky 'script-kiddy' way. Looking out at the community of different projects it becomes clear that everyone is pretty lost as to what design principles should be used for certain tasks, so every repo has its own way of doing things (I know personal style occurs in other languages, but commonalities are much less recognizable in R projects). It's tragic that such a large community uses it.
  - lordgroff 4y ago
    Yep, this is a mark of someone that's never used R but has heard a lot of incredibly ill informed criticism around it. One look of dplyr code over pandas would of course disabuse anyone of the notion that R is trash and the tragedy is Python will in the current state never have anything like that. That's the advantage of the language being influenced by Lisp vs not.
    - tomrod 4y ago
      I've heavily used R several times. I agree that it is a trash language and that, outside that many frontier academic ideas are available and some plotting preferences are solidly prescriptive, it should be thrown into the trash bin. Python, Julia when it gets its druthers for TTFP, Octave, Fortran, C, and eventually Rust. These are the tools I've found in use over and over and over again across business, government, and non-profits. Everywhere R is used by the org I have seen major gaps in capacity to deliver specifically because R doesn't scale well.
      - nerdponx 4y ago
        Try to separate the language from its standard library. Neither one is "trash". I agree that the standard library is what you might call "a chaotic disorganized mess".
        tomrod 4y ago
        I'm not emotionally invested in tools so am happy to identify the user experience and operational experience as "trash." "Trash", despite its connotations of lacking value, is really just a chaotic disorganized mess of something made by artifice with dubious reclaim/reuse/recycle value. Being a subjective assessment, it is natural that one person's trash is a treasure to another.
        nerdponx 4y ago
        I take issue with your implication that I'm emotionally invested in something when I shouldn't be. You are free to dislike R and not use it, but to claim that it's "trash" is to wrongly disavow its usefulness for the many people that do find it useful, and to cast aspersions on the judgement of all those people.
        tomrod 4y ago
        Hey, I apologize here, my point on emotional investment was that I, personally, am not emotionally invested in it and did not mean to cast aspersions at you for your defense of the language nor at people who have preferences for it. Specifically I meant that I'm comfortable enough in my understanding of the language to classify it and it's standard library as better in the garbage bin relative to alternatives available. It's fine that people like it. What's good about it isn't unique, and what's unique about it isn't that great. And there are certainly switching costs for some orgs to consider.
  - jmt_ 4y ago
    Trash language is a bit harsh. I'm not sure I would try to put an R project into production or build a huge project with it but, at the very least, R/R Studio was the best scientific calculator I've ever used. Was particularly great during college
  - jjgreen 4y ago
    It's not trash, it's functional, look: x <- 3

whatever1 4y ago

I have never seen a worse documented library. Initially I thought that they were lazy, now I realize that it cannot be documented because it is a total mess of a library held together with tape. Close second is the plotly library.

__mharrison__ 4y ago
Genuinely curious what you have against the Pandas documentation. It has some of the best docstrings I've seen. (I also wrote a Pandas book or two... So there's that)
- chaps 4y ago
  Docstrings are one thing, but functionality discovery, picking up from scratch, troubleshooting, etc are... not fun, nor easy with the documentation. If you know it well already and use it a lot it's easier to forgive its documentation faults since you can waive off the problems as "that's just learning something new". But for a lot of people who use it infrequently its documentation is a frustrating mess. Simple problems turn into significant time sinks of trying to find which page of the documentation to look at. A lot of issues are made worse by shit-awful interop between libraries that claim to fully support dayaframes, but often fail in non-obvious ways... meaning back to the documentation mines. I'd argue that because there's a market for a single author to write two books about it is indicative of documentation problems.
  - __mharrison__ 4y ago
    Fair enough. I'm highly biased and my recent book is the most popular Pandas book currently, so it is evidence that folks prefer opinionated documentation. However, I always though the 10 minutes to Pandas page was decent for getting started. I picked up Polars recently and thought it was more difficult than Pandas because there wasn't any quick intro docs. What projects have great introductory docs for you? Also, I am curious to learn more about the specifics of interop libraries you are referring to. Learning a new tool is generally a challenge. I think another challenge with a lot of data tools is that non-programmers tend to be the major audience. I make my living teaching "non-programmers" how to use these tools. That said, I always teach "go to the docstrings and stay in your environment (to not break flow) if you can." The pydata docstrings are better than most, including Python (the language).

dekhn 4y ago

What bothers me the most is the egregious data types for any argument. If it's a string, do this. If it's a list, do that. If it's a dictionary of lists, do this other thing. No, I want you to force me to provide my data in the right way and raise a noisy exception if I don't.

nerdponx 4y ago
Series and DataFrame have "alternate constructors" for this purpose, and the loc/iloc accessors give you a bit more control. I agree that the magic type auto-detection is a bit too magical and sloppy, but you have to realize that data analysts and scientists have historically been incredibly sloppy programmers who wanted as much magic as possible. It's only in recent years that researchers have begun to value some amount of discipline in their research code.