4 ms·

> but most data scientists are not programmers This is the crux of the problem with R and why R is increasingly blacklisted at large orgs. It attracts non-prog

> but most data scientists are not programmers This is the crux of the problem with R and why R is increasingly blacklisted at large orgs. It attracts non-programmers which may have been okay 5 years ago but is no longer acceptable. With the exception of some engineering powerhouses hiring pure research PhDs to write R code, the trend established over the last 2 years is that fewer and fewer employers are hiring data scientists that aren't programmers. There are too many candidates who know data science and can also do data engineering and even generalist SE tasks. Non-programmer data scientists are not competitive in the industry anymore except that small top-end research niche that doesn't exist in most orgs. Which brings us back to the fact that R was a successful niche language that allowed non-programmers to write models, but that's simply not enough anymore. Businesses want models that can be plugged into production pipelines, models that can scale without needing a dedicated team to re-implement them, and they want staff who do engineering in addition to whatever it is they specialise in. Virtually all data scientists graduating today are programmers, and pretty good ones. Candidates who only know R can't compete against them. > Lack of a magrittr style infix operator, though seemingly minor, actually emerges as a real pain point once you become accustomed to using So you'd agree that you fall into the 'I prefer the syntax' bucket then? I don't really see any arguments against Python in your comment. Funnily enough, it's trivial to implement a pipe style operator in Python and there's at least two popular libraries for that.

fock 7y ago
and yet all these businesses run excel in production. I'd rather implement R code (with localizreed variables and dumb algorithms, so there's something funny in it) than an excel-spreadsheet. But somehow there's a difference...
- stoops 7y ago
  They really don't. I've been at such an org as described by OP. I've owned a system of production R/Excel and it was migrated to cloud + python + ETL over 4 years. The places where Excel are used are fairly appropriate. Way downstream, for simple tasks.
  - starfallg 7y ago
    A lot of hard science is still done on the back of Excel, only begrudgingly adopting a data science mindset as the instruments produce more and more data. Data science is more than just streaming data, data lakes and machine learning. Visual programming platforms like Knime is the next step for these teams, and then onto something like RStudio as they complete the transition towards employing data science in their pipelines.
darkhorn 7y ago
>It attracts non-programmers If employers are hiring non-statisticians as data scientists then the problem is the employers.
- missosoup 7y ago
  Stats and programming aren't mutually exclusive. The current generation of DS graduates are strong in both. Job applicants who only know R and have no grasp of SE are increasingly less and less competitive. I don't expect there'll be any market for them in another 5 years.
  - darkhorn 7y ago
    In Statistics they teach programming. For example we have studied one semester C, one semester OOP (C++), one semester SQL and relational database design. All these were must courses. Other than these also R, Matlab, Minitab, SPSS and SAS. All of my classmates knew programming. It is stupid to think that in this age a statistician won't be able to write a program. How you are going to make analyses of a population census in a country as a statistician? Statistical packages do not always provide all you need. Sometimes you need to transform a data. Sometimes you need to check/validate a data. Sometimes you need to query from an X location. Sometimes you need to pipe through some process. Few people from our department wrote R packages that didn't exist (new statistical analyses).
CapmCrackaWaka 7y ago
I actually use R mostly because of its data.table package. It is much faster and more concise than pandas, which is a nightmare to work with. Sure, you can get the job done in pandas, but you often have to wait ~10x longer for your commands to run and sometimes, I simply cannot use pandas at all because I run out of memory. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/ People are usually pretty surprised when I take the stance that R is faster than Python for the things most people actually care about, which is data manipulation and model building.
hyperbovine 7y ago
> R is increasingly blacklisted at large orgs. Eh, I call BS. Names and sources please. I know for a fact that R is used at all of FAANG and about a bazillion other "large orgs" too. I'm sure it's true that R is not used for customer-facing "web scale" products, but then again neither is any other language except for like two. Being good at programming is useful skill, but so is being good at statistics, and they are not interchangeable. "Productionizing a model" is not the only show in town when it comes to data analysis. Many programmers know shockingly little statistics. An equally large number of really strong statisicians prefer R, for good reasons. Orgs who simply refuse to hire those people do so at their peril.

fock 7y ago

and yet all these businesses run excel in production. I'd rather implement R code (with localizreed variables and dumb algorithms, so there's something funny in it) than an excel-spreadsheet. But somehow there's a difference...

stoops 7y ago
They really don't. I've been at such an org as described by OP. I've owned a system of production R/Excel and it was migrated to cloud + python + ETL over 4 years. The places where Excel are used are fairly appropriate. Way downstream, for simple tasks.
- starfallg 7y ago
  A lot of hard science is still done on the back of Excel, only begrudgingly adopting a data science mindset as the instruments produce more and more data. Data science is more than just streaming data, data lakes and machine learning. Visual programming platforms like Knime is the next step for these teams, and then onto something like RStudio as they complete the transition towards employing data science in their pipelines.

darkhorn 7y ago

>It attracts non-programmers If employers are hiring non-statisticians as data scientists then the problem is the employers.

missosoup 7y ago
Stats and programming aren't mutually exclusive. The current generation of DS graduates are strong in both. Job applicants who only know R and have no grasp of SE are increasingly less and less competitive. I don't expect there'll be any market for them in another 5 years.
- darkhorn 7y ago
  In Statistics they teach programming. For example we have studied one semester C, one semester OOP (C++), one semester SQL and relational database design. All these were must courses. Other than these also R, Matlab, Minitab, SPSS and SAS. All of my classmates knew programming. It is stupid to think that in this age a statistician won't be able to write a program. How you are going to make analyses of a population census in a country as a statistician? Statistical packages do not always provide all you need. Sometimes you need to transform a data. Sometimes you need to check/validate a data. Sometimes you need to query from an X location. Sometimes you need to pipe through some process. Few people from our department wrote R packages that didn't exist (new statistical analyses).

CapmCrackaWaka 7y ago

I actually use R mostly because of its data.table package. It is much faster and more concise than pandas, which is a nightmare to work with. Sure, you can get the job done in pandas, but you often have to wait ~10x longer for your commands to run and sometimes, I simply cannot use pandas at all because I run out of memory. https://h2oai.github.io/db-benchmark/ https://h2oai.github.io/db-benchmark/ People are usually pretty surprised when I take the stance that R is faster than Python for the things most people actually care about, which is data manipulation and model building.

hyperbovine 7y ago

> R is increasingly blacklisted at large orgs. Eh, I call BS. Names and sources please. I know for a fact that R is used at all of FAANG and about a bazillion other "large orgs" too. I'm sure it's true that R is not used for customer-facing "web scale" products, but then again neither is any other language except for like two. Being good at programming is useful skill, but so is being good at statistics, and they are not interchangeable. "Productionizing a model" is not the only show in town when it comes to data analysis. Many programmers know shockingly little statistics. An equally large number of really strong statisicians prefer R, for good reasons. Orgs who simply refuse to hire those people do so at their peril.