5 ms·

If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway? Even if R has better "versions" of pandas/matplotlib (not co

by pc86 5y ago

If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway? Even if R has better "versions" of pandas/matplotlib (not conceding that point) it's not exactly central to the job of data science.

civilized 5y ago
As a working data scientist I'd say it's completely the opposite: a good tabular data manipulation package is the single most valuable tool in my tool box. And R's packages (either data.table or dplyr) are definitely way better than pandas. There's no comparison. I would be hard-pressed to find a working data scientist whose definition of data science is "that thing you do with sklearn, Deep Learning and Numpy".
- jstx1 5y ago
  > "Data science is that thing where you do sklearn, Deep Learning and Numpy" is not a working data scientist's perspective. It could be. It's such a broad job title and it looks so different across different companies and teams that the main tool for one data scientist might be something that another data scientist never has to touch. Different data science jobs prioritise different tools, that's all.
  - civilized 5y ago
    Right, so defining data science as 90% sklearn+DL+numpy is just as silly as saying that it's 90% table manipulation. That's exactly my point. Still, if anyone here has managed to find a data science job in which tabular data management is not a sizable piece of what you do, I'd like to know some details!
    - mrtranscendence 5y ago
      I imagine there are data scientists who operate primarily on unstructured rather than tabular data. Part of my current job involves stuff like text classification, and it's not that difficult to imagine someone for whom that's a more sizable proportion of their day-to-day. Still, my suspicion -- at least from my corner of data science -- is that such individuals are rare, and that most data scientists do make use of tabular data more often than not.
      - civilized 5y ago
        I totally get what you mean - I would suspect that when you work with unstructured data, tabular data manipulation is maybe 20-40% of what you do, and when you work with structured data, it's more like 60-80%.
    - _Wintermute 5y ago
      I worked as a datascientist for a couple of years and tabular data was a very small part of my job. I spent far more time with image-analysis and JSON, both of which I found R sucks at.
- pletnes 5y ago
  Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me.
  - civilized 5y ago
    Of course, my post doesn't imply such a silly statement.
  - disgruntledphd2 5y ago
    > Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me. Most problems are mostly tabular, IME. I completely agree that text, images and video are much, much better handled by Python (that's why I use and know both).
bllguo 5y ago
maybe we are casualties of the vague definition of "data science," but in my experience numpy is too low-level for most of what I consider DS, and pandas/matplotlib are _much_ more central than sklearn or pytorch. Even if your definition only encompasses deep learning research, surely plotting is still indispensable? I'll also add my vote for the superiority of data.table and ggplot2 to any Python alternatives. the bloat and verbosity of pandas is a daily struggle
- MichaelRazum 5y ago
  Just curious. In which way is data.table superior to pandas? Really interested about it! From my personal experience pandas is just sometimes a bit slow.
  - mrtranscendence 5y ago
    I'm more a dplyr man myself, but data.table is much faster than pandas, most noticeably IMO when reading large files. It's also extremely succinct if you're into that sort of thing (though I find it a bit obfuscated). pandas is a lot of things, but "fast" and "concise" are not two of them.
    - MichaelRazum 5y ago
      Got it. Regarding fast you have something like Vaex on python side (but not sure how fast it realy is). For me I had with pandas the most issues using it's multiindex.
      - mrtranscendence 5y ago
        > For me I had with pandas the most issues using it's multiindex. Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them. > Regarding fast you have something like Vaex on python sid I've never used Vaex, but I've used datatable (https://github.com/h2oai/datatable https://github.com/h2oai/datatable) and polars (https://github.com/pola-rs/polars https://github.com/pola-rs/polars). Polars is my favorite API, but datatable was faster at reading data (Polars was faster in execution). I'll have to give Vaex a try at some point.
disgruntledphd2 5y ago
> If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway? Not really, tbh. Most of my jobs (even when the primary output was models) require spending a _lot_ of time data wrangling and plotting. R is much, much better for this kind of exploratory work. But if I need to integrate with bigger systems (as I normally do), there's a stronger push for Python to reduce complexity and make it easier for SE's to understand and maintain (some of) the code.

by pc86 5y ago

civilized 5y ago

As a working data scientist I'd say it's completely the opposite: a good tabular data manipulation package is the single most valuable tool in my tool box. And R's packages (either data.table or dplyr) are definitely way better than pandas. There's no comparison. I would be hard-pressed to find a working data scientist whose definition of data science is "that thing you do with sklearn, Deep Learning and Numpy".

jstx1 5y ago
> "Data science is that thing where you do sklearn, Deep Learning and Numpy" is not a working data scientist's perspective. It could be. It's such a broad job title and it looks so different across different companies and teams that the main tool for one data scientist might be something that another data scientist never has to touch. Different data science jobs prioritise different tools, that's all.
- civilized 5y ago
  Right, so defining data science as 90% sklearn+DL+numpy is just as silly as saying that it's 90% table manipulation. That's exactly my point. Still, if anyone here has managed to find a data science job in which tabular data management is not a sizable piece of what you do, I'd like to know some details!
  - mrtranscendence 5y ago
    I imagine there are data scientists who operate primarily on unstructured rather than tabular data. Part of my current job involves stuff like text classification, and it's not that difficult to imagine someone for whom that's a more sizable proportion of their day-to-day. Still, my suspicion -- at least from my corner of data science -- is that such individuals are rare, and that most data scientists do make use of tabular data more often than not.
    - civilized 5y ago
      I totally get what you mean - I would suspect that when you work with unstructured data, tabular data manipulation is maybe 20-40% of what you do, and when you work with structured data, it's more like 60-80%.
  - _Wintermute 5y ago
    I worked as a datascientist for a couple of years and tabular data was a very small part of my job. I spent far more time with image-analysis and JSON, both of which I found R sucks at.
pletnes 5y ago
Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me.
- civilized 5y ago
  Of course, my post doesn't imply such a silly statement.
- disgruntledphd2 5y ago
  > Tabular data is great for many usecases, but saying that image, audio, and video analysis is not data science seems like a weird variant of gatekeeping to me. Most problems are mostly tabular, IME. I completely agree that text, images and video are much, much better handled by Python (that's why I use and know both).

bllguo 5y ago

maybe we are casualties of the vague definition of "data science," but in my experience numpy is too low-level for most of what I consider DS, and pandas/matplotlib are _much_ more central than sklearn or pytorch. Even if your definition only encompasses deep learning research, surely plotting is still indispensable? I'll also add my vote for the superiority of data.table and ggplot2 to any Python alternatives. the bloat and verbosity of pandas is a daily struggle

MichaelRazum 5y ago
Just curious. In which way is data.table superior to pandas? Really interested about it! From my personal experience pandas is just sometimes a bit slow.
- mrtranscendence 5y ago
  I'm more a dplyr man myself, but data.table is much faster than pandas, most noticeably IMO when reading large files. It's also extremely succinct if you're into that sort of thing (though I find it a bit obfuscated). pandas is a lot of things, but "fast" and "concise" are not two of them.
  - MichaelRazum 5y ago
    Got it. Regarding fast you have something like Vaex on python side (but not sure how fast it realy is). For me I had with pandas the most issues using it's multiindex.
    - mrtranscendence 5y ago
      > For me I had with pandas the most issues using it's multiindex. Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them. > Regarding fast you have something like Vaex on python sid I've never used Vaex, but I've used datatable (https://github.com/h2oai/datatable https://github.com/h2oai/datatable) and polars (https://github.com/pola-rs/polars https://github.com/pola-rs/polars). Polars is my favorite API, but datatable was faster at reading data (Polars was faster in execution). I'll have to give Vaex a try at some point.

disgruntledphd2 5y ago

> If you're doing data science aren't sklearn, DL, and numpy getting you 90% of the way there anyway? Not really, tbh. Most of my jobs (even when the primary output was models) require spending a _lot_ of time data wrangling and plotting. R is much, much better for this kind of exploratory work. But if I need to integrate with bigger systems (as I normally do), there's a stronger push for Python to reduce complexity and make it easier for SE's to understand and maintain (some of) the code.