4 ms·

What data-analysis package do you prefer over pandas? I've used pandas since circa 2015, and lately have been trying to become fluent in R and the Tidyverse, bu

What data-analysis package do you prefer over pandas? I've used pandas since circa 2015, and lately have been trying to become fluent in R and the Tidyverse, but it's been hard so far to unlearn the deeply ingrained python/pandas patterns.

wodenokoto 7y ago
Start with R for data science (book by Hadley, available online for free) Tidy verse assumes tidy data. If you are not working with tidy data, it is unlikely to be a big help. Most data can probably be thought of as tidy. Remember that any and every operation on a data frame returns a data frame, so unlike chaining in Pandas, you never have to worry if a method you want to use belongs to a series or a data frame, or if your method is returning a series or a data frame. Select() selects columns, filter() selects rows. This never changes unlike the [] which means different thing depending on if it is used on a data frame (which you are not guaranteed to be served after calling a method on data frame in pandas!) on a series or using the .loc or .iloc methods. There is no index, instead you just filter on rows. Pandas comes with a ton of build in utilities which the tidyverse doesn’t, mostly because R is already full of functions you can easily apply across columns. But particularly pandas date handling functions are really cool
- EForEndeavour 7y ago
  That's reassuring: I'm (slowly) working through Hadley Wickam's r4ds online book, which is just fantastic. Thanks for the tidyverse tips!
smabie 7y ago
There’s nothing for Python except Pandas. I came from a FP and static typing background before I moved into ML/quant finance and initially found Pandas incredibly difficult to reason about. The tool is designed for scientists who know nothing about how a library should be designed or how a program should be structured. There’s a lot of dynamic stuff in Pandas that while making things easier for scientists make things a lot more difficult for CS people. Same thing with numpy and scipy, and other data science libraries. My absolute favorite DataFrame library is saddle (for Scala), which I helped write at my old quant job. Very FP oriented and an absolute pleasure to use. Though maybe it’s no surprise that I like something I worked on. An incomplete list of things that I dislike about Pandas are: Too many parameters and knobs for each function Inconsistenty between inplace and copying operations Unintuitive function names compared to FP Too much magic in how things work Functions and parameters accept a wide range of types in order to make things “just work.” Lots of non-orthogonal convenience functions that do mostly the same thing I’m not familiar with how “normal” Python is written, but I suspect a lot of the problems come from the abuse of dynamic typing. Dynamic typing allows you to just add more and more levels of crap without actually changing your data/type model. I think there’s a lot of value in “correct” APIs, vs convenient ones. That being said, Pandas is extremely powerful, and usually very succinct. Maybe not as nice as kdb+/q (nothing really compares for time-series data), but still pretty good.

wodenokoto 7y ago

Start with R for data science (book by Hadley, available online for free) Tidy verse assumes tidy data. If you are not working with tidy data, it is unlikely to be a big help. Most data can probably be thought of as tidy. Remember that any and every operation on a data frame returns a data frame, so unlike chaining in Pandas, you never have to worry if a method you want to use belongs to a series or a data frame, or if your method is returning a series or a data frame. Select() selects columns, filter() selects rows. This never changes unlike the [] which means different thing depending on if it is used on a data frame (which you are not guaranteed to be served after calling a method on data frame in pandas!) on a series or using the .loc or .iloc methods. There is no index, instead you just filter on rows. Pandas comes with a ton of build in utilities which the tidyverse doesn’t, mostly because R is already full of functions you can easily apply across columns. But particularly pandas date handling functions are really cool

EForEndeavour 7y ago
That's reassuring: I'm (slowly) working through Hadley Wickam's r4ds online book, which is just fantastic. Thanks for the tidyverse tips!

smabie 7y ago

There’s nothing for Python except Pandas. I came from a FP and static typing background before I moved into ML/quant finance and initially found Pandas incredibly difficult to reason about. The tool is designed for scientists who know nothing about how a library should be designed or how a program should be structured. There’s a lot of dynamic stuff in Pandas that while making things easier for scientists make things a lot more difficult for CS people. Same thing with numpy and scipy, and other data science libraries. My absolute favorite DataFrame library is saddle (for Scala), which I helped write at my old quant job. Very FP oriented and an absolute pleasure to use. Though maybe it’s no surprise that I like something I worked on. An incomplete list of things that I dislike about Pandas are: Too many parameters and knobs for each function Inconsistenty between inplace and copying operations Unintuitive function names compared to FP Too much magic in how things work Functions and parameters accept a wide range of types in order to make things “just work.” Lots of non-orthogonal convenience functions that do mostly the same thing I’m not familiar with how “normal” Python is written, but I suspect a lot of the problems come from the abuse of dynamic typing. Dynamic typing allows you to just add more and more levels of crap without actually changing your data/type model. I think there’s a lot of value in “correct” APIs, vs convenient ones. That being said, Pandas is extremely powerful, and usually very succinct. Maybe not as nice as kdb+/q (nothing really compares for time-series data), but still pretty good.