3 ms·

My whole work consists in manipulating arrays of numbers, mostly in python, and I never found any use for pandas. Whenever I receive some code that uses pandas

My whole work consists in manipulating arrays of numbers, mostly in python, and I never found any use for pandas. Whenever I receive some code that uses pandas, it is easy to remove this dependency without much ado (it was not really necessary for anything). Can anybody point me to a reasonable use case of pandas? I mean, besides printing a matrix with lines of alternating colors.

cerved 7y ago
Lots of built in statistical stuff and powerful visualization makes exploring datasets easy
- enriquto 7y ago
  > Lots of built in statistical stuff and powerful visualization makes exploring datasets easy I see. For linear algebra stuff it does not offer anything essential. You rarely see a matrix as a "dataset".
siddboots 7y ago
My whole work consists of manipulating arrays of numbers, mostly in python. I use pandas by default, and treat numpy as a specialised tool to be brought out when the use-case necessitates. Some big things for me are a) timeseries methods that just work e.g. df.resample, b) and multi-indices with stack/unstack etc., c) best I/O of any python data library (e.g. read_csv, read_parquet, read_h5, read_sql, etc)
wodenokoto 7y ago
Interesting. How do you do group by's and joins?
- enriquto 7y ago
  I don't know exactly what you mean, but numpy's hstack and vstack seem to be enough for me.
  - wodenokoto 7y ago
    SQL-like joins. Fx, joining the sum of sales by store ID from the sales table with the table of store names on the store ID. That is a major reason why people use pandas.
Nimitz14 7y ago
I think it's useful for when your feature columns have names. Otherwise yeah not really.
throwaway287391 7y ago
I feel the same way. I do ML research and every time I'm provided a Pandas dataframe, the first thing I do before I try to work with it (after getting over my vague feeling of annoyance) is convert it to a format I understand -- e.g. a numpy array, a dict of numpy arrays, etc. I find the pandas API very unintuitive from an idiomatic Python perspective.* And it's not that I've never used SQL (I've written some small websites/apps that use it), I just haven't encountered a situation in my ML work where I've needed joins etc. to the point where I'd care to learn a whole new API for them. I gather it's very useful and extremely well loved for certain types of work though. But "the usability of Pandas" definitely doesn't ring true for me, let alone the implication that numpy is relatively unusable...if you've written lots of idiomatic Python working with its native data structures, NumPy will come very naturally, whereas Pandas is like a whole new language. * I have to call `at` instead of using Python indexing to get a row. wat. If I try to iterate over it like "for x in df", x is a column name rather than a row. wat.
smabie 7y ago
The multi-index functionality is very nice for financial time-series data. I can easily resample from days to months, filter rows, etc. It’s good for columns that are of different types. I don’t really like using it though, Julia’s DataFrames are easier to use, more performant, and more integrated into the language.

cerved 7y ago

Lots of built in statistical stuff and powerful visualization makes exploring datasets easy

enriquto 7y ago
> Lots of built in statistical stuff and powerful visualization makes exploring datasets easy I see. For linear algebra stuff it does not offer anything essential. You rarely see a matrix as a "dataset".

siddboots 7y ago

My whole work consists of manipulating arrays of numbers, mostly in python. I use pandas by default, and treat numpy as a specialised tool to be brought out when the use-case necessitates. Some big things for me are a) timeseries methods that just work e.g. df.resample, b) and multi-indices with stack/unstack etc., c) best I/O of any python data library (e.g. read_csv, read_parquet, read_h5, read_sql, etc)

wodenokoto 7y ago

Interesting. How do you do group by's and joins?

enriquto 7y ago
I don't know exactly what you mean, but numpy's hstack and vstack seem to be enough for me.
- wodenokoto 7y ago
  SQL-like joins. Fx, joining the sum of sales by store ID from the sales table with the table of store names on the store ID. That is a major reason why people use pandas.

Nimitz14 7y ago

I think it's useful for when your feature columns have names. Otherwise yeah not really.

throwaway287391 7y ago

I feel the same way. I do ML research and every time I'm provided a Pandas dataframe, the first thing I do before I try to work with it (after getting over my vague feeling of annoyance) is convert it to a format I understand -- e.g. a numpy array, a dict of numpy arrays, etc. I find the pandas API very unintuitive from an idiomatic Python perspective.* And it's not that I've never used SQL (I've written some small websites/apps that use it), I just haven't encountered a situation in my ML work where I've needed joins etc. to the point where I'd care to learn a whole new API for them. I gather it's very useful and extremely well loved for certain types of work though. But "the usability of Pandas" definitely doesn't ring true for me, let alone the implication that numpy is relatively unusable...if you've written lots of idiomatic Python working with its native data structures, NumPy will come very naturally, whereas Pandas is like a whole new language. * I have to call `at` instead of using Python indexing to get a row. wat. If I try to iterate over it like "for x in df", x is a column name rather than a row. wat.

smabie 7y ago

The multi-index functionality is very nice for financial time-series data. I can easily resample from days to months, filter rows, etc. It’s good for columns that are of different types. I don’t really like using it though, Julia’s DataFrames are easier to use, more performant, and more integrated into the language.