7 ms·

Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, aft

Hehe used to do R IMO you are right about ggplot but I strongly disagree about pandas. I fing love it. Would love to understand you troubles with it though, after using it for 4 years daily mabye I can offer some perspective ;)

mrtranscendence 5y ago
I don't mean to disparage pandas, which is a library that does a lot of things fairly well. But as an API for data manipulation I find it very verbose and it doesn't mesh with a "functional" way of thinking about applying transformations. Generally, I've even preferred Spark to pandas, though it's hardly less verbose. Coming from R, it's much slower than data.table and nowhere near as slick and discoverable as dplyr. Its system of indices is a pain that I'd rather not deal with at all (and, indeed, I can't think of another data frame library that relies on them). I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ... Handles time series really well, though. Recently I've been using polars (https://github.com/pola-rs/polars https://github.com/pola-rs/polars). As an API I much, much prefer it to pandas, and it's a lot faster. Comes at the cost of not using numpy under the hood, so you can't just toss a polars data frame into a sklearn model.
- disgruntledphd2 5y ago
  Agreed on your major points. That being said: > I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ... This is also default in R, with row numbers (like I have ever needed them). To be fair, it's gotten better since people stopped putting important information in rownames. Polars looks interesting, thanks for the recommendation!
- deshpand 5y ago
  < I hate finding CSVs that other data scientists Ideally you should be using the parquet format which will use the binary format, preserve column types and indexes [df.to_parquet(<file>); df = pd.read_parquet(<file>)] You can get away from a lot of problems by simply avoiding text files
disgruntledphd2 5y ago
It reminds me of base R from 2010, and i thought dplyr had driven a stake through the heart of those approaches. More generally, the API is large, all-consuming and not consistent. sklearn is best in class here, I rarely need to look things up whereas the pandas docs autocomplete in my browser after one or two characters.
andreareina 5y ago
I run into pandas edge cases all the time. pd.concat() failing on empty sequences (just let me specify a default for that case please); .squeeze() not letting me say, "squeeze down to a series but not a scalar"; .groupby().apply() returning different types depending on how many groups/rows per group there are... it's fine when you know exactly what you have but it's hard using it in a pipeline with that needs to be agnostic about whether there's zero, one, or many data (datums?).
baron_harkonnen 5y ago
Pandas indexing system is overly complex and I've never personally benefited from that. To start with there are __getitem__, loc and iloc approaches to accessing values. If your library constantly has to warn users that "you might being something wrong, read the docs!" that should be a warning sign that you don't have the correct level of abstraction. R has a much more sane api and assumptions about when you want to access a value by reference (which is almost always) and by value. Then when doing basic operations like "group by" you end up excessively elaborate indexes that are in my experience useless and always need to be manually squashed to something coherent. It's a common joke for me that whenever even a seasoned Pandas user cries out "gaarrr! why isn't this working!?" I just reply "have you tried reset_index?"... this works in a frighteningly large number of cases.

mrtranscendence 5y ago

I don't mean to disparage pandas, which is a library that does a lot of things fairly well. But as an API for data manipulation I find it very verbose and it doesn't mesh with a "functional" way of thinking about applying transformations. Generally, I've even preferred Spark to pandas, though it's hardly less verbose. Coming from R, it's much slower than data.table and nowhere near as slick and discoverable as dplyr. Its system of indices is a pain that I'd rather not deal with at all (and, indeed, I can't think of another data frame library that relies on them). I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ... Handles time series really well, though. Recently I've been using polars (https://github.com/pola-rs/polars https://github.com/pola-rs/polars). As an API I much, much prefer it to pandas, and it's a lot faster. Comes at the cost of not using numpy under the hood, so you can't just toss a polars data frame into a sklearn model.

disgruntledphd2 5y ago
Agreed on your major points. That being said: > I hate finding CSVs that other data scientists have created from pandas, because they invariably include the index ... This is also default in R, with row numbers (like I have ever needed them). To be fair, it's gotten better since people stopped putting important information in rownames. Polars looks interesting, thanks for the recommendation!
deshpand 5y ago
< I hate finding CSVs that other data scientists Ideally you should be using the parquet format which will use the binary format, preserve column types and indexes [df.to_parquet(<file>); df = pd.read_parquet(<file>)] You can get away from a lot of problems by simply avoiding text files

disgruntledphd2 5y ago

It reminds me of base R from 2010, and i thought dplyr had driven a stake through the heart of those approaches. More generally, the API is large, all-consuming and not consistent. sklearn is best in class here, I rarely need to look things up whereas the pandas docs autocomplete in my browser after one or two characters.

andreareina 5y ago

I run into pandas edge cases all the time. pd.concat() failing on empty sequences (just let me specify a default for that case please); .squeeze() not letting me say, "squeeze down to a series but not a scalar"; .groupby().apply() returning different types depending on how many groups/rows per group there are... it's fine when you know exactly what you have but it's hard using it in a pipeline with that needs to be agnostic about whether there's zero, one, or many data (datums?).

baron_harkonnen 5y ago

Pandas indexing system is overly complex and I've never personally benefited from that. To start with there are __getitem__, loc and iloc approaches to accessing values. If your library constantly has to warn users that "you might being something wrong, read the docs!" that should be a warning sign that you don't have the correct level of abstraction. R has a much more sane api and assumptions about when you want to access a value by reference (which is almost always) and by value. Then when doing basic operations like "group by" you end up excessively elaborate indexes that are in my experience useless and always need to be manually squashed to something coherent. It's a common joke for me that whenever even a seasoned Pandas user cries out "gaarrr! why isn't this working!?" I just reply "have you tried reset_index?"... this works in a frighteningly large number of cases.