5 ms·

This really grinds my gears too. There's something about the pandas API that makes it impossible for me to do basic ops without tedious manual browsing to get i

This really grinds my gears too. There's something about the pandas API that makes it impossible for me to do basic ops without tedious manual browsing to get inplace or index arguments right... assignments and conditionals are needlessly verbose too. Pyspark on the other hand just sticks in my brain, somehow. Chained pyspark method calls looks much neater.

ziml77 6y ago
Setting inplace=True isn't too bad, but I definitely have had many issues with working with indexes in Pandas. I don't understand why they didn't design it so that the index can be referenced like any other columns. It overcomplicates things like having to know the subtle difference between join() and merge().
- nojito 6y ago
  Setting Inplace=True is not recommended and should be used with caution https://github.com/pandas-dev/pandas/issues/16529 https://github.com/pandas-dev/pandas/issues/16529
  - ziml77 6y ago
    I've never seen anything about inplace being a poor idea to use. Is that documented anywhere or is that ticket the only info about it?
    - Peritract 6y ago
      It's not particularly front-and-centre, but it's all over various discussion boards if you go looking for it directly. I'd like the debate to be more visible, personally, particularly whenever the argument gets deprecated for a particular method. Essentially, `inplace=True` rarely actually saves memory, and causes problems if you like chaining things together. The people who maintain the library/populate the discussion boards are generally pro-chaining, so `inplace` is slowly and quietly on its way out.
      - ziml77 6y ago
        I wish it wasn't something you had to go looking for. I never would have thought that it would be an issue. If you do Google searches for inplace, you get no results on the first page discouraging its use. The documentation should clearly say if the inplace argument causes an internal copy, because the availability of it implies that it doesn't. I've used inplace many times with Pandas because I've had code that I know is working with large amounts of data and I've thought "well I probably shouldn't chain these and cause tons of unnecessary allocations just to have prettier code".
disgruntledphd2 6y ago
Pandas is just a bad API, to be honest. I know base-R very, very well (which was one of the inspirations, I believe) and I still spend most of my time looking stuff up. It's such a shame that python doesn't have a better DF library.
- shankr 6y ago
  I also switched from R to Python/pandas. I remember always being frustrated with pandas since it tries to emulate data.frame, but then just does its own thing without being consistent.
- EForEndeavour 6y ago
  I've used pandas regularly for the past ~5 years and find its API intuitive enough not to complain. I can write and read decently long pandas chained expressions fluently, but I barely know any R. Am I unwittingly a hostage of an inferior API and don't know what I'm missing?
  - disgruntledphd2 6y ago
    Yes. Base R is OK, but dplyr is magical. For instance, integer indexing in base R is df[row,col] rather than the iloc pandas stuff. plot, print and summary (and generic function OOP more generally is really underappreciated). Python is a better programming language, but R is a better data analysis environment. And dplyr is an incredibly fluent DSL for doing data analysis (not quite as good for modelling though). Seriously, I read the original vignette for dplyr in late 2013/early 2014 and within two weeks I'd switched most of my new analytical code over to it. So very, very good. Less idea-impedance match than any other environment, in my experience.
    - shankr 6y ago
      I was actually using data.table The syntax can be bit cryptic, but you get used to it.
  - deleted 6y ago
    [deleted]

ziml77 6y ago

Setting inplace=True isn't too bad, but I definitely have had many issues with working with indexes in Pandas. I don't understand why they didn't design it so that the index can be referenced like any other columns. It overcomplicates things like having to know the subtle difference between join() and merge().

nojito 6y ago
Setting Inplace=True is not recommended and should be used with caution https://github.com/pandas-dev/pandas/issues/16529 https://github.com/pandas-dev/pandas/issues/16529
- ziml77 6y ago
  I've never seen anything about inplace being a poor idea to use. Is that documented anywhere or is that ticket the only info about it?
  - Peritract 6y ago
    It's not particularly front-and-centre, but it's all over various discussion boards if you go looking for it directly. I'd like the debate to be more visible, personally, particularly whenever the argument gets deprecated for a particular method. Essentially, `inplace=True` rarely actually saves memory, and causes problems if you like chaining things together. The people who maintain the library/populate the discussion boards are generally pro-chaining, so `inplace` is slowly and quietly on its way out.
    - ziml77 6y ago
      I wish it wasn't something you had to go looking for. I never would have thought that it would be an issue. If you do Google searches for inplace, you get no results on the first page discouraging its use. The documentation should clearly say if the inplace argument causes an internal copy, because the availability of it implies that it doesn't. I've used inplace many times with Pandas because I've had code that I know is working with large amounts of data and I've thought "well I probably shouldn't chain these and cause tons of unnecessary allocations just to have prettier code".

disgruntledphd2 6y ago

Pandas is just a bad API, to be honest. I know base-R very, very well (which was one of the inspirations, I believe) and I still spend most of my time looking stuff up. It's such a shame that python doesn't have a better DF library.

shankr 6y ago
I also switched from R to Python/pandas. I remember always being frustrated with pandas since it tries to emulate data.frame, but then just does its own thing without being consistent.
EForEndeavour 6y ago
I've used pandas regularly for the past ~5 years and find its API intuitive enough not to complain. I can write and read decently long pandas chained expressions fluently, but I barely know any R. Am I unwittingly a hostage of an inferior API and don't know what I'm missing?
- disgruntledphd2 6y ago
  Yes. Base R is OK, but dplyr is magical. For instance, integer indexing in base R is df[row,col] rather than the iloc pandas stuff. plot, print and summary (and generic function OOP more generally is really underappreciated). Python is a better programming language, but R is a better data analysis environment. And dplyr is an incredibly fluent DSL for doing data analysis (not quite as good for modelling though). Seriously, I read the original vignette for dplyr in late 2013/early 2014 and within two weeks I'd switched most of my new analytical code over to it. So very, very good. Less idea-impedance match than any other environment, in my experience.
  - shankr 6y ago
    I was actually using data.table The syntax can be bit cryptic, but you get used to it.
- deleted 6y ago
  [deleted]