13 ms·

R Passes SAS in Scholarly Use

193 points by sndean 10y ago

benbenolson 10y ago
Just noticed that this was posted by my old (half) boss! Finally! This is very encouraging, that such an excellent free software package is in such high demand. From what I've used it for, it worked very well. It's great for quickly creating nice-looking graphs and plots.
dredmorbius 10y ago
1. This is a good place for use of log or semi-log plots. 2. How do the authors unambiguously search for 'R'? Monocharacter language names are difficult search keys. (C, B, S, R)
- haddr 10y ago
  There are some keywords that could help: R Lang and Rstats for instance
- thebelal 10y ago
  1. There is a log plot at the end of the article 2. The author posted the exact search terms used for all languages in an earlier post [0] [0]: http://r4stats.com/articles/how-to-search-for-data-science-articles/ http://r4stats.com/articles/how-to-search-for-data-science-a...
  - dredmorbius 10y ago
    Thanks, on both points.
- LionessLover 10y ago
  To 2: When I look for just "R" even in an anonymous window (so it should not use my history) I get as the first suggestion a link to https://www.r-project.org/ https://www.r-project.org/ - the home of R. What else is there for that letter - that is equally popular? "R" is "hip" and trending. Microsoft not too long ago started a big push into the R space and now regularly generates headlines around the system, accelerating the trend even more.
  - jhbadger 10y ago
    Even weirder, a Google search for "xlispstat" seems to bring up more R hits that don't even mention xlispstat than actual xlispstat ones. Some weird algorithm is associating R and xlispstat as relating to statistics and because R is much more popular these days, prioritizing R over xlispstat.
    - yoplait_ 10y ago
      it's not the search engine's fault, it's the contents. R is associated to xlispstat as the credible alternative. E.g. Jan de Leeuw JStat's paper. Also the author of xlispstat is one main contributor to R.
sillysaurus3 10y ago
Mathematica isn't being used at all? That's surprising. Mathematica is wonderful. I wonder what's holding it back? It doesn't seem to have a package manager. Could it be that simple?
- benbenolson 10y ago
  Yes, it's extremely easy to link to packages in R, and even to include C++ programs to make your R package run faster. Mathematica, on the other hand, is a large proprietary package that involves a bit of effort to install and use. I'm not sure about what statistical features it has, but perhaps they're not as developed as R's.
- hoka 10y ago
  price?
- detaro 10y ago
  I'm not sure if Mathematica is on the radar if you are looking for "just" a stats package, it can do way more. I have no idea how it compares price-wise to SAS?
  - oso2k 10y ago
    For a single licensed installation, over $7k per year for SAS. Every add-on/package increases price as well. What we're charged is inline with [0]. [0] https://www.quora.com/How-much-does-SAS-cost https://www.quora.com/How-much-does-SAS-cost
    - boomskats 10y ago
      Shameless self promotion, but not every package increases the price. We maintain a free (GPLv3) library for JS/SAS that lets you build nice user interfaces to your programs/workflows using modern frameworks like Angular or React. Check out github.com/boemska/h54s. I wouldn't normally post it like this, but SAS comes up so rarely that I figured if you're on HN and you use SAS, then you'll probably be interested. The more the merrier.
thefastlane 10y ago
no mention of Incanter . . . is it really that niche?
- Bootvis 10y ago
  I think it is, I'd be interested to hear user stories.
- IndianAstronaut 10y ago
  It is still quite underdeveloped when compared to R. Might have a use case since it is on the JVM.
- jhbadger 10y ago
  Is it still alive? I remember being excited about it when it came out initially but then things seemed to slow and then stop.
uptownfunk 10y ago
Some reasons I love / use R: Plenty of free high quality documentation and learning materials around R (just read anything by Hadley) Package manager. Super easy to find, install, and start using packages. Open source / Free Large community of users Extensive usage by the stats community. (If a new algorithm comes out, chances R there will be an R implementation) Easy to build and share your own packages via Github. Easy to link C++ code to your packages. ---------------------------------------------- I love R, but something about how the language feels syntactically, it's not as pleasurable programming wise compared to something like the Python data stack. But with all of the above advantages, I don't see myself switching to anything else in the future for my data science work, unless I have a really pressing need to. The other thing is that the language is so damn popular that the useR conference was sold out in pre-reg. rounds.. Seriously guys, stop using and learning about R so I can get in the conference....
- nol13 10y ago
  Basically this. Certainly not the worst language (SAS), but intangibly less pleasurable than python or most other common languages I've used. Maybe that's because I'm not a 'real' stats person though and ~ notation still takes me a minute to grok. And there is no challenger on he horizon for it's dominance in available packages. And at least it's not SAS. My theories: A) Stat people use R because because they don't know any better B) Stat people use R because code monkeys use python C) Stat people use R because stat people use R (probably this)
  - minimaxir 10y ago
    D) Stat people use R and Python, depending on what best fits the use case.
    - nol13 10y ago
      Ya but the only use case for R over python is because some package is written in it.
  - hadley 10y ago
aabajian 10y ago
I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data frames, Java's ArrayLists, etc. I'd advise any new graduate student to learn how to plug data into traditional programs, but save yourself a headache and perform your data manipulation in Python. Eventually you can take the leap and do the analysis in Python as well.
- moyok 10y ago
  Julia is great too for this kind of work
- jacobolus 10y ago
  The even bigger advantage of python is everything else: web scraping, interfacing with weird APIs, hitting the database, consuming and emitting obscure formats, parsing text, calling operating system services, and a million other things. That and much much better abstractions for building larger systems out of reusable components and much better tooling for serious software engineering. R or Matlab can be fine at exploratory data analysis, making charts, running miscellaneous bits of non-programmer-grad-student code found on the internet, etc., but as soon as you want to do anything other than data analysis, they quickly become annoying.
  - minimaxir 10y ago
    If you haven't looked at the R ecosystem in awhile, there is a package for each of those use cases. (scraping/API: rvest; database: dplyr; parsing text: stringr, etc). Yes, Hadley Wickham is primarily responsible for the popularity of R.
    - jacobolus 10y ago
      We’re talking about an order of magnitude difference in number of packages (82096 on PyPI vs. 8551 on CRAN) and their maturity, and such a naïve metric probably undersells the difference in variety of use cases. If you picked 100 random production python projects out of a hat, no more than a small handful of them would be remotely appropriate to build using R. And that's entirely fine. R is great at being a quick and dirty statistical analysis language for people doing data exploration.
rsrsrs86 10y ago
Am I right to believe that there is no way that proprietary scientific software can keep up with open source?
- LeifCarrotson 10y ago
  No. Some have the benefit of proprietary modules (FPGA toolchains), some have large libraries of pre-entered and organized data (Mathematica), and some have early access to hardware (LabView, CUDA). Programming languages, perhaps, are less vulnerable to these issues. And perhaps open source could beat these applications eventually, given perfect competition. But we're not in that world, unfortunately.
  - kpil 10y ago
    I think that open source programming languages will always win in the long run, since the target customer base knows how to program and extend the tools.
    - gaius 10y ago
      I'm not sure that logically follows. What %age of Python users actually know the underlying C well enough to make changes to the language? Even the number who know how to write bindings is tiny overall.
      - kpil 10y ago
        I'm fairly sure that most of them would be able to wing it if they had to. C isn't exactly that special. In all events, the percentage of users skilled in software development must be higher than among users of tools not related to software development.
- Hondor 10y ago
  For simulation it's not even close. Open source can't do user interfaces. Not a problem for programming languages, but for simulation at least, there's a slew of powerful but unusable open source software made by professors and then there's expensive proprietary ones with nice front ends that save enormous amount of time for the users.
erik_landerholm 10y ago
R? Everyone doing most anything interesting has been using Python for a while.
- hugh4 10y ago
  But what language do true Scotsmen use?
- Jweb_Guru 10y ago
  I suppose there is no interesting science outside of computers?
gaius 10y ago
One thing that interests me is language power, vs experience. Let's say you had 1 year experience in language X. Language Y comes along that is better in some way. In another year, would you be happier and more productive with 2 years experience of X, or one year of Y? I sometimes think with the churn of languages, no-one really gets deeply enough into one to really leverage it.
kensai 10y ago
The success of R in Statistics (in respect to Python, etc) was that it was thought from the beginning with Statisticians and their specific needs and approaches in mind. As much as I appreciate Python, it is a general purpose programming language adapted to Statisticians needs, not the other way around. R has many issues, but if you speak to Statisticians you will hear that its the closest thing they have to their own way of doing things.
- lottin 10y ago
  This is absolutely true, and actually learning R is an excellent way of learning statistics.
  - Myrmornis 10y ago
    and a very inadvisable way to learn programming. (But I agree with what you said.)
mikeskim 10y ago
The way I use Python in machine learning is quite different from how many others in competitive ML use Python. I use Python purely for Python 2.7 with Pypy and try not to touch or use numpy,scipy,pandas,etc. R's data.table is possibly faster than Python's numpy/scipy/pandas. I think anyone claiming Python because of numpy/scipy/pandas is really being mislead. You should be using Python in spite of the need to rely upon numpy/scipy/pandas. If you really need numpy/scipy/pandas just use R and data.table which is amazingly fast. I think Python is really great because of Pypy and the strength of the standard Python library.
pollitos 10y ago
R is really LISP with syntactic sugar and bindings to well respected high-performance FORTRAN matrix and math optimization codes. http://librestats.com/2011/08/27/how-much-of-r-is-written-in-r/ http://librestats.com/2011/08/27/how-much-of-r-is-written-in... It's great for bleeding edge scientific research. The results of many languages don't always match for advanced algorithms, but the open source nature of R, makes it easier to identify the problem areas. The R-core interpreter does have a number of deficiencies. (R is based on S-language specification that left wiggle room from the 70s.) General purpose programming and data wrangling/engineering is best handled in other programming idioms.
- jhbadger 10y ago
  People say that, but I'd prefer the actual LISP syntax then (being a fan of xlispstat back in the day). I'm surprised nobody has created a "Lisp-flavored R" analogous to Erlang's LFE or Python's Hy.
  - hadley 10y ago
    Most R users aren't programmers, and a LISP-y interface, while powerful, tends to be intimidating.
  - cwyers 10y ago
    I'd prefer something more like TypeScript for R, where you can gradually move over but you get better tooling. I'd also ask for a new standard library but I think Hadley is basically doing that.
- hadley 10y ago
  I would love to know what you mean by data wrangling because I think R has a lot of good tools for it.
  - pollitos 10y ago
    For example, reshaping JSON to the format an intricate R function expects. Appreciate the great work with (d)plyr and similar packages, but it's still work and overhead. Combined with some inefficiencies/quirks in base r functions (does ifelse() still evaluate twice?) it's easier to go with a widely used and respected package in a general purpose language; Nokogiri for example. For data engineering, consider there is not a maintaned R package for a web-client, and asynchronous programming is weak.
mdjt 10y ago
This is great news! It excites me to see the continued support of open-source and well-maintained programming languages in academics. (Someone may have said this already, but there is no way I'm reading through all the "Python vs R" BS to find out)
daix 10y ago
>Note that the decline in the number of articles that used SPSS or SAS is not balanced by the increase in the other software shown in this particular graph. Is Machine Learning cooling down?
Malarkey73 10y ago
I'm not totally sure whether this analysis captures the true extent which R vs SAS vs SPSS is used. If I use R for a plot, or a simple bit of regression, or anova, or even cross-validation. I don't reference it in a paper. I only cite it if there is a package designed for a particular type of data (e.g. a Bioconductor package) or something a bit more esoteric (e.g. apcluster). About 95% of the work is data munging and - sorry Hadley - I don't cite dplyr, purrr, magrittr etc... However I have notice that in clinical trial or small social science papers simple analyses of this type are often cited as being done in SPSS or SAS. I think this just reflects the fact that non specialist data analysts are more likely to cite SAS or SPSS for simple procedures such as graphs or anova as an appeal to authority. So I reckon the data may reflect a trend but tells us little about the true levels.
- ececconi 10y ago
  This is especially true in undergrad papers.
- jsprogrammer 10y ago
  Shouldn't your full source be available; which would explicitly record your dependencies?
  - griffinmichl 10y ago
    Source code really should be available, but it almost never is. The peer review process is, in my opinion, quite flawed. While your paper's high level content gets reviewed, no one actually looks at your code and data to ensure that you didn't forget to carry the one. Your analysis could be totally wrong, but reviewers only review what you say you did, not what you actually did.
    - chrisamiller 10y ago
      Any paper I peer review better have source code available or they will hear about it. That said, yes, there isn't time (or funding) to actually re-run the entire analysis.
dekhn 10y ago
EDIT: I am corrected in regards to the SAS routines statement; see the reply. A few comments. I worked in pharma and the FDA specifically requires a number of SAS routines- specific function calls- to be used when doing drug studies/clinical trials. R can't replace SAS in those cases without massive effort because the FDA is slow and conservative and people like to have validated results. I think the writing was on the wall for SAS when this article came out: http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html?pagewanted=all http://www.nytimes.com/2009/01/07/technology/business-comput... The SAS spokesperson said: """Anne H. Milley, director of technology product marketing at SAS. She adds, “We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”""" to which a senior employee of Boeing pointed out that every jet they build uses R as an integral part of the design process. I think that had to be an "oh shit" moment for SAS, where they realized their strong position in stats was going to start to erode.
- hadley 10y ago
  That is not true. The FDA uses R internally, and there is no requirement that you must use any specific software tool. See https://www.r-project.org/doc/R-FDA.pdf https://www.r-project.org/doc/R-FDA.pdf for more details
  - dekhn 10y ago
    OK, so my employers were wrong! http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html """Despite some mistaken conceptions in the pharmaceutical industry, SAS is not required to be used for clinical trials. This origin of this fallacy is probably related to the fact that data must be submitted in the XPT "transport format" (which was originally created by SAS). This data format is now an open standard: XPT files can be read into R with the standard read.xport function, and exported from R with the write.xport function in the SASxport package. (And if you have legacy data in other SAS formats, there's a handy SAS macro to export XPT files.)""" Thanks for clearning that up.
jiiam 10y ago
Good. * Rant mode: On Maybe in 30 years they will also learn a true programming language and stop producing undocumented, unusable, unportable, underdeveloped libraries for research level tools and technologies. Outside the world of Neural Network it is a complete disaster, and the NN landscape is at an acceptable level only because of big companies, surely not thanks to the researchers. And the reason, of course, is that most researchers refuse to think of themselves as "software developer" and use these arcane languages which might be good for prototyping but lack power when it comes to shipping a real product (which might also be a tool for other researchers to use). At least they're not using Matlab where everything breaks as soon as you change machine. * Rant mode: Off
- noelsusman 10y ago
  I mean, I won't argue against having better code and documentation, but it's not really our job to ship a real product. Shipping well documented, easily usable, ultra portable, well developed libraries takes a fuckload of time, resources, and expertise that we don't have. Our primary job is to ship ideas. It would be awesome if every project I did ended up with a nice, polished piece of software, but that's not what I get paid to do. I would be fired if I tried to do that.
  - jiiam 10y ago
    Fair enough. But, speaking as a researcher, I often find myself reading through hundreds of lines of code and rebuilding routines from scratch in order to reproduce and expand on what others did in their works. However, I was very harsh, and of course I wouldn't find viable to expect production ready code, but something moderately portable could come handy. Of course, as you said, a researcher doesn't have the time to build a well developed library. As a solution, my University is considering the idea of hiring a dedicated developer whose job would be to maintain libraries. I really hope this to happen.

193 points by sndean 10y ago

Just noticed that this was posted by my old (half) boss! Finally! This is very encouraging, that such an excellent free software package is in such high demand. From what I've used it for, it worked very well. It's great for quickly creating nice-looking graphs and plots.

dredmorbius 10y ago

1. This is a good place for use of log or semi-log plots. 2. How do the authors unambiguously search for 'R'? Monocharacter language names are difficult search keys. (C, B, S, R)

haddr 10y ago
There are some keywords that could help: R Lang and Rstats for instance
thebelal 10y ago
1. There is a log plot at the end of the article 2. The author posted the exact search terms used for all languages in an earlier post [0] [0]: http://r4stats.com/articles/how-to-search-for-data-science-articles/ http://r4stats.com/articles/how-to-search-for-data-science-a...
- dredmorbius 10y ago
  Thanks, on both points.
LionessLover 10y ago
To 2: When I look for just "R" even in an anonymous window (so it should not use my history) I get as the first suggestion a link to https://www.r-project.org/ https://www.r-project.org/ - the home of R. What else is there for that letter - that is equally popular? "R" is "hip" and trending. Microsoft not too long ago started a big push into the R space and now regularly generates headlines around the system, accelerating the trend even more.
- jhbadger 10y ago
  Even weirder, a Google search for "xlispstat" seems to bring up more R hits that don't even mention xlispstat than actual xlispstat ones. Some weird algorithm is associating R and xlispstat as relating to statistics and because R is much more popular these days, prioritizing R over xlispstat.
  - yoplait_ 10y ago
    it's not the search engine's fault, it's the contents. R is associated to xlispstat as the credible alternative. E.g. Jan de Leeuw JStat's paper. Also the author of xlispstat is one main contributor to R.

sillysaurus3 10y ago

Mathematica isn't being used at all? That's surprising. Mathematica is wonderful. I wonder what's holding it back? It doesn't seem to have a package manager. Could it be that simple?

benbenolson 10y ago
Yes, it's extremely easy to link to packages in R, and even to include C++ programs to make your R package run faster. Mathematica, on the other hand, is a large proprietary package that involves a bit of effort to install and use. I'm not sure about what statistical features it has, but perhaps they're not as developed as R's.
hoka 10y ago
price?
detaro 10y ago
I'm not sure if Mathematica is on the radar if you are looking for "just" a stats package, it can do way more. I have no idea how it compares price-wise to SAS?
- oso2k 10y ago
  For a single licensed installation, over $7k per year for SAS. Every add-on/package increases price as well. What we're charged is inline with [0]. [0] https://www.quora.com/How-much-does-SAS-cost https://www.quora.com/How-much-does-SAS-cost
  - boomskats 10y ago
    Shameless self promotion, but not every package increases the price. We maintain a free (GPLv3) library for JS/SAS that lets you build nice user interfaces to your programs/workflows using modern frameworks like Angular or React. Check out github.com/boemska/h54s. I wouldn't normally post it like this, but SAS comes up so rarely that I figured if you're on HN and you use SAS, then you'll probably be interested. The more the merrier.

thefastlane 10y ago

no mention of Incanter . . . is it really that niche?

Bootvis 10y ago
I think it is, I'd be interested to hear user stories.
IndianAstronaut 10y ago
It is still quite underdeveloped when compared to R. Might have a use case since it is on the JVM.
jhbadger 10y ago
Is it still alive? I remember being excited about it when it came out initially but then things seemed to slow and then stop.

uptownfunk 10y ago

Some reasons I love / use R: Plenty of free high quality documentation and learning materials around R (just read anything by Hadley) Package manager. Super easy to find, install, and start using packages. Open source / Free Large community of users Extensive usage by the stats community. (If a new algorithm comes out, chances R there will be an R implementation) Easy to build and share your own packages via Github. Easy to link C++ code to your packages. ---------------------------------------------- I love R, but something about how the language feels syntactically, it's not as pleasurable programming wise compared to something like the Python data stack. But with all of the above advantages, I don't see myself switching to anything else in the future for my data science work, unless I have a really pressing need to. The other thing is that the language is so damn popular that the useR conference was sold out in pre-reg. rounds.. Seriously guys, stop using and learning about R so I can get in the conference....

nol13 10y ago
Basically this. Certainly not the worst language (SAS), but intangibly less pleasurable than python or most other common languages I've used. Maybe that's because I'm not a 'real' stats person though and ~ notation still takes me a minute to grok. And there is no challenger on he horizon for it's dominance in available packages. And at least it's not SAS. My theories: A) Stat people use R because because they don't know any better B) Stat people use R because code monkeys use python C) Stat people use R because stat people use R (probably this)
- minimaxir 10y ago
  D) Stat people use R and Python, depending on what best fits the use case.
  - nol13 10y ago
    Ya but the only use case for R over python is because some package is written in it.
- hadley 10y ago

aabajian 10y ago

I think Python is the biggest hidden gem in statistics. It's had a tremendous impact on machine learning and algorithm development, yet traditional statisticians still rely on SAS/R/Stata/MATLAB. All of these languages have libraries that produce the same results, the difficulty is mangling the data into the correct input format. Python's list comprehensions are much, much easier to use than MATLAB matrices, R's data frames, Java's ArrayLists, etc. I'd advise any new graduate student to learn how to plug data into traditional programs, but save yourself a headache and perform your data manipulation in Python. Eventually you can take the leap and do the analysis in Python as well.

moyok 10y ago
Julia is great too for this kind of work
jacobolus 10y ago
The even bigger advantage of python is everything else: web scraping, interfacing with weird APIs, hitting the database, consuming and emitting obscure formats, parsing text, calling operating system services, and a million other things. That and much much better abstractions for building larger systems out of reusable components and much better tooling for serious software engineering. R or Matlab can be fine at exploratory data analysis, making charts, running miscellaneous bits of non-programmer-grad-student code found on the internet, etc., but as soon as you want to do anything other than data analysis, they quickly become annoying.
- minimaxir 10y ago
  If you haven't looked at the R ecosystem in awhile, there is a package for each of those use cases. (scraping/API: rvest; database: dplyr; parsing text: stringr, etc). Yes, Hadley Wickham is primarily responsible for the popularity of R.
  - jacobolus 10y ago
    We’re talking about an order of magnitude difference in number of packages (82096 on PyPI vs. 8551 on CRAN) and their maturity, and such a naïve metric probably undersells the difference in variety of use cases. If you picked 100 random production python projects out of a hat, no more than a small handful of them would be remotely appropriate to build using R. And that's entirely fine. R is great at being a quick and dirty statistical analysis language for people doing data exploration.

rsrsrs86 10y ago

Am I right to believe that there is no way that proprietary scientific software can keep up with open source?

LeifCarrotson 10y ago
No. Some have the benefit of proprietary modules (FPGA toolchains), some have large libraries of pre-entered and organized data (Mathematica), and some have early access to hardware (LabView, CUDA). Programming languages, perhaps, are less vulnerable to these issues. And perhaps open source could beat these applications eventually, given perfect competition. But we're not in that world, unfortunately.
- kpil 10y ago
  I think that open source programming languages will always win in the long run, since the target customer base knows how to program and extend the tools.
  - gaius 10y ago
    I'm not sure that logically follows. What %age of Python users actually know the underlying C well enough to make changes to the language? Even the number who know how to write bindings is tiny overall.
    - kpil 10y ago
      I'm fairly sure that most of them would be able to wing it if they had to. C isn't exactly that special. In all events, the percentage of users skilled in software development must be higher than among users of tools not related to software development.
Hondor 10y ago
For simulation it's not even close. Open source can't do user interfaces. Not a problem for programming languages, but for simulation at least, there's a slew of powerful but unusable open source software made by professors and then there's expensive proprietary ones with nice front ends that save enormous amount of time for the users.

erik_landerholm 10y ago

R? Everyone doing most anything interesting has been using Python for a while.

hugh4 10y ago
But what language do true Scotsmen use?
Jweb_Guru 10y ago
I suppose there is no interesting science outside of computers?

gaius 10y ago

One thing that interests me is language power, vs experience. Let's say you had 1 year experience in language X. Language Y comes along that is better in some way. In another year, would you be happier and more productive with 2 years experience of X, or one year of Y? I sometimes think with the churn of languages, no-one really gets deeply enough into one to really leverage it.

kensai 10y ago

The success of R in Statistics (in respect to Python, etc) was that it was thought from the beginning with Statisticians and their specific needs and approaches in mind. As much as I appreciate Python, it is a general purpose programming language adapted to Statisticians needs, not the other way around. R has many issues, but if you speak to Statisticians you will hear that its the closest thing they have to their own way of doing things.

lottin 10y ago
This is absolutely true, and actually learning R is an excellent way of learning statistics.
- Myrmornis 10y ago
  and a very inadvisable way to learn programming. (But I agree with what you said.)

mikeskim 10y ago

The way I use Python in machine learning is quite different from how many others in competitive ML use Python. I use Python purely for Python 2.7 with Pypy and try not to touch or use numpy,scipy,pandas,etc. R's data.table is possibly faster than Python's numpy/scipy/pandas. I think anyone claiming Python because of numpy/scipy/pandas is really being mislead. You should be using Python in spite of the need to rely upon numpy/scipy/pandas. If you really need numpy/scipy/pandas just use R and data.table which is amazingly fast. I think Python is really great because of Pypy and the strength of the standard Python library.

pollitos 10y ago

R is really LISP with syntactic sugar and bindings to well respected high-performance FORTRAN matrix and math optimization codes. http://librestats.com/2011/08/27/how-much-of-r-is-written-in-r/ http://librestats.com/2011/08/27/how-much-of-r-is-written-in... It's great for bleeding edge scientific research. The results of many languages don't always match for advanced algorithms, but the open source nature of R, makes it easier to identify the problem areas. The R-core interpreter does have a number of deficiencies. (R is based on S-language specification that left wiggle room from the 70s.) General purpose programming and data wrangling/engineering is best handled in other programming idioms.

jhbadger 10y ago
People say that, but I'd prefer the actual LISP syntax then (being a fan of xlispstat back in the day). I'm surprised nobody has created a "Lisp-flavored R" analogous to Erlang's LFE or Python's Hy.
- hadley 10y ago
  Most R users aren't programmers, and a LISP-y interface, while powerful, tends to be intimidating.
- cwyers 10y ago
  I'd prefer something more like TypeScript for R, where you can gradually move over but you get better tooling. I'd also ask for a new standard library but I think Hadley is basically doing that.
hadley 10y ago
I would love to know what you mean by data wrangling because I think R has a lot of good tools for it.
- pollitos 10y ago
  For example, reshaping JSON to the format an intricate R function expects. Appreciate the great work with (d)plyr and similar packages, but it's still work and overhead. Combined with some inefficiencies/quirks in base r functions (does ifelse() still evaluate twice?) it's easier to go with a widely used and respected package in a general purpose language; Nokogiri for example. For data engineering, consider there is not a maintaned R package for a web-client, and asynchronous programming is weak.

mdjt 10y ago

This is great news! It excites me to see the continued support of open-source and well-maintained programming languages in academics. (Someone may have said this already, but there is no way I'm reading through all the "Python vs R" BS to find out)

daix 10y ago

>Note that the decline in the number of articles that used SPSS or SAS is not balanced by the increase in the other software shown in this particular graph. Is Machine Learning cooling down?

Malarkey73 10y ago

I'm not totally sure whether this analysis captures the true extent which R vs SAS vs SPSS is used. If I use R for a plot, or a simple bit of regression, or anova, or even cross-validation. I don't reference it in a paper. I only cite it if there is a package designed for a particular type of data (e.g. a Bioconductor package) or something a bit more esoteric (e.g. apcluster). About 95% of the work is data munging and - sorry Hadley - I don't cite dplyr, purrr, magrittr etc... However I have notice that in clinical trial or small social science papers simple analyses of this type are often cited as being done in SPSS or SAS. I think this just reflects the fact that non specialist data analysts are more likely to cite SAS or SPSS for simple procedures such as graphs or anova as an appeal to authority. So I reckon the data may reflect a trend but tells us little about the true levels.

ececconi 10y ago
This is especially true in undergrad papers.
jsprogrammer 10y ago
Shouldn't your full source be available; which would explicitly record your dependencies?
- griffinmichl 10y ago
  Source code really should be available, but it almost never is. The peer review process is, in my opinion, quite flawed. While your paper's high level content gets reviewed, no one actually looks at your code and data to ensure that you didn't forget to carry the one. Your analysis could be totally wrong, but reviewers only review what you say you did, not what you actually did.
  - chrisamiller 10y ago
    Any paper I peer review better have source code available or they will hear about it. That said, yes, there isn't time (or funding) to actually re-run the entire analysis.

dekhn 10y ago

EDIT: I am corrected in regards to the SAS routines statement; see the reply. A few comments. I worked in pharma and the FDA specifically requires a number of SAS routines- specific function calls- to be used when doing drug studies/clinical trials. R can't replace SAS in those cases without massive effort because the FDA is slow and conservative and people like to have validated results. I think the writing was on the wall for SAS when this article came out: http://www.nytimes.com/2009/01/07/technology/business-computing/07program.html?pagewanted=all http://www.nytimes.com/2009/01/07/technology/business-comput... The SAS spokesperson said: """Anne H. Milley, director of technology product marketing at SAS. She adds, “We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”""" to which a senior employee of Boeing pointed out that every jet they build uses R as an integral part of the design process. I think that had to be an "oh shit" moment for SAS, where they realized their strong position in stats was going to start to erode.

hadley 10y ago
That is not true. The FDA uses R internally, and there is no requirement that you must use any specific software tool. See https://www.r-project.org/doc/R-FDA.pdf https://www.r-project.org/doc/R-FDA.pdf for more details
- dekhn 10y ago
  OK, so my employers were wrong! http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html http://blog.revolutionanalytics.com/2012/06/fda-r-ok.html """Despite some mistaken conceptions in the pharmaceutical industry, SAS is not required to be used for clinical trials. This origin of this fallacy is probably related to the fact that data must be submitted in the XPT "transport format" (which was originally created by SAS). This data format is now an open standard: XPT files can be read into R with the standard read.xport function, and exported from R with the write.xport function in the SASxport package. (And if you have legacy data in other SAS formats, there's a handy SAS macro to export XPT files.)""" Thanks for clearning that up.

jiiam 10y ago

Good. * Rant mode: On Maybe in 30 years they will also learn a true programming language and stop producing undocumented, unusable, unportable, underdeveloped libraries for research level tools and technologies. Outside the world of Neural Network it is a complete disaster, and the NN landscape is at an acceptable level only because of big companies, surely not thanks to the researchers. And the reason, of course, is that most researchers refuse to think of themselves as "software developer" and use these arcane languages which might be good for prototyping but lack power when it comes to shipping a real product (which might also be a tool for other researchers to use). At least they're not using Matlab where everything breaks as soon as you change machine. * Rant mode: Off

noelsusman 10y ago
I mean, I won't argue against having better code and documentation, but it's not really our job to ship a real product. Shipping well documented, easily usable, ultra portable, well developed libraries takes a fuckload of time, resources, and expertise that we don't have. Our primary job is to ship ideas. It would be awesome if every project I did ended up with a nice, polished piece of software, but that's not what I get paid to do. I would be fired if I tried to do that.
- jiiam 10y ago
  Fair enough. But, speaking as a researcher, I often find myself reading through hundreds of lines of code and rebuilding routines from scratch in order to reproduce and expand on what others did in their works. However, I was very harsh, and of course I wouldn't find viable to expect production ready code, but something moderately portable could come handy. Of course, as you said, a researcher doesn't have the time to build a well developed library. As a solution, my University is considering the idea of hiring a dedicated developer whose job would be to maintain libraries. I really hope this to happen.