7 ms·

Data Analysts Captivated by Power of R

82 points by kalvin 18y ago

gruseom 18y ago
Meanwhile the creator of R wants to return to a Lisp-based statistics environment: http://books.google.com/books?id=8Cf16JkKz30C&pg=PA21&lpg=PA21 http://books.google.com/books?id=8Cf16JkKz30C&pg=PA21&#3...
- jderick 18y ago
  Another interesting paper hidden behind the academic firewall.
  - Anon84 18y ago
    At least in the physics world people just post everything to arXiv and their personal websites. This way you only need to pay attention to a couple of places to keep up to date.
    - brent 18y ago
      Even in certain areas of computer science (specifically, machine learning) nearly all recent articles are open. I find in most cases that both statisticians and computer scientists tend to post links to pdfs or ps on their websites as well.
    - scott_s 18y ago
      As I said in another thread, most computer science papers are on authors' webpages. Then this one comes along - but it looks like the authors are stat people. I don't know what their culture is - and what kind of copyright agreements they sign. I can access the paper because my school subscribes. If anyone wants to read it, figure out how to email me and I'll send a copy. While searching, I did find a brief email from Ihaka talking about the jump to Lisp: https://stat.ethz.ch/pipermail/r-devel/2008-May/049501.html https://stat.ethz.ch/pipermail/r-devel/2008-May/049501.html
      - gaius 18y ago
        Thanks :-)
- steveblgh 18y ago
  Lush is a LISP-style environment for scientific computing: http://lush.sourceforge.net/ http://lush.sourceforge.net/
- mwexler 18y ago
  Statistics and Lisp have a long, long history together. Before R took off, many academic statisticians were using Lisp-Stat, Luke Tierney's tool from UMN. Luke has now been spending time helping R, and it shows in how it is starting to adopt more of a Lisp mentality. The referenced paper also shows the Lisp background from R (and S) history. R syntax can be confusing enough; Lisp really has a learning effort such that most analysts just don't have the time to invest in it unless they were trained in it early on.
- twopoint718 18y ago
  Maxima (http://maxima.sourceforge.net/ http://maxima.sourceforge.net/) is a nice CAS, sort of similar to Maple, IIRC. It is written in Lisp but hides that from the interface layer.
scott_s 18y ago
I am baffled that a programming language is getting coverage in the NYT. I'd like to figure out what, exactly, it is about this one that merits mainstream coverage, but I'm afraid I've been at the office too long and I think my perspective on the matter is permanently skewed. As an aside, I used R briefly for a econometrics/philosophy course I took. I recognized immediately it was a powerful, functional language. What I wonder, though, is if the scientific programming libraries in Python might eventually be a better environment. Surely there must be some R users here who can comment.
- jderick 18y ago
  Economics is hot right now?
  - daniel-cussen 18y ago
    In my opinion it is. I have anticipated this downturn for the rich historical perspective on depressions it will offer.
- Anon84 18y ago
  if the scientific programming libraries in Python might eventually be a better environment. Well... there is R.py http://rpy.sourceforge.net/ http://rpy.sourceforge.net/ to give you most (if not all) the functionality of R in conjunction with all the scipy goodness.
- showerst 18y ago
  Having only very minimal experience with R, I get the impression that while python is fantastic for programmers, and the tools are probably close to or better than R's (no idea about library support), it's still a programming language, and requires one to think and structure things like a programmer. R, at least in the beginning, comes across more like a powerful set of Excel formulas, so I think a non-programmer might pick it up faster without having good programming form. (Not to say one is more powerful than the other, or that R programmers aren't as good as Python programmers, etc, I just think it's a difference in community viewpoints)
tokenadult 18y ago
"The co-creators of R express satisfaction that such companies profit from the fruits of their labor and that of hundreds of volunteers." That's an interesting reaction from the first designers of the program.
zandorg 18y ago
"We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet." - Ugh, SAS isn't about science, it's about administration.
- bkj123 18y ago
  What do you mean by "SAS isn't about science, it's about administration"
  - zandorg 18y ago
    I meant SAP, but that or SAS is about 'business intelligence', which means running an organisation - staffing, payroll, etc, but certainly not jet engines. My Aunt programs in SAP for a University. So SAP determines how businesses are run, and are paid for the privelege.
    - neilc 18y ago
      SAP and SAS are completely different. SAS is for science (well, for statistical programming in general); SAS has nothing to do with SAP, which is for back-office stuff. The article and the quote are about statistics packages like SAS.
- michaelneale 18y ago
  What an idiot. There would be lots of freeware involved if not on the aircraft. And that is preferable to the alternative in many cases.
  - aneesh 18y ago
    Well, for starters, the video screens on the seatbacks of 777s (and other aircraft) are running some flavor of Linux. But that's not a counterexample to her argument, per se, since the video screens aren't critical to the plane's operation.
    - 18y ago
wesm 18y ago
having programmed R on the job for some heavy statistics, I will say this: good for quick analyses but burdened by legacy functionality from the S+ days. I switched to Python/NumPy and rewrote all the R code I had, could not be happier with the results. Of course, you have to create your own data structures if you want something like R's data frame, but at least you have a rich language to do that with. however, if you need to do anything systematic, do NOT use R, bugs are elusive and extremely tedious to debug
- yters 18y ago
  I have used R for a number of minor projects. I really like its functional programming design, but its syntax can be obtuse. It's preferable to MatLab, IMHO. What do you consider to be the legacy aspects that weigh it down? What do you like better about NumPy? While I'm asking questions, have you tried Sage? If so, what do you think of it as a meta package of mathematical software? Also, here are interesting links I've found comparing R and similar statistical programs. Sorry, don't know much about Numpy's capabilities. R equivalent in speed to Matlab: http://www.sciviews.org/benchmark/index.html http://www.sciviews.org/benchmark/index.html Someone's research for a good data analysis language: http://www.cs.ubc.ca/~murphyk/Software/which_language.html http://www.cs.ubc.ca/~murphyk/Software/which_language.html
  - wesm 18y ago
    One of R's big benefits is the huge amount of statistical functionality, for example numerous different quantilization algorithms. If you are working with a lot of heterogeneous data in R it becomes a real headache. Merging data frames seems like it should work like you think it should but if one of your sets of keys (strings) are 'factors' (what I am calling 'legacy S+ functionality', I'm sure they're useful for many algorithms), you'll end up with garbage. There's a hack you can put in your code ('options(stringsAsFactors=FALSE)') which alleviates some of this but in general aligning data I found to be a huge pain. If you're running regressions this is pretty important haven't tried sage but have heard good things. NumPy is a good alternative because it's extremely well implemented and has consistent behavior across the board. Extensibility (with Fortran, Cython/Pyrex, C/C++) is clean and easy. Never thought I'd write Fortran 77 code being born quite a few years after '77 but it's an easy way to speed up simple procedural algorithms 50x or more.
jessep 18y ago
Interesting. My girlfriend is a statistician for the WHO and they definitely still use SAS, at least in her area (calculating global burden of disease). I'm going to ask her if anyone there uses R.
- jessep 18y ago
  she says apparently there's a big debate going on at the WHO about whether everything should be done in R. currently they use R, Stata, and SAS.
manny 18y ago
I can't believe nobody here has mentioned PDL, the Perl Data Language: http://pdl.perl.org http://pdl.perl.org Admittedly, i think R and PDL do different things... (I have never played with R).
- draegtun 18y ago
  Looks like PDL == NumPy (http://news.ycombinator.com/item?id=363159 http://news.ycombinator.com/item?id=363159) There are CPAN modules to directly use R from Perl (for eg... R::* & Statistics::R). I have a stats friend who's been singing the praises of R & PDL for donkey years. /I3az/
bbgm 18y ago
In the life science space, R dominates research informatics. A large chunk of molecular profiling methods and techniques use R, or quite often the Bioconductor package, http://www.bioconductor.org/ http://www.bioconductor.org/ (from Gentleman's group). Most commercial bioinformatics apps also implement a number of methods using R and provide ability to implement R-based classifiers, etc. In the clinical space, it's all SAS. Pretty much the de facto standard.
- stcredzero 18y ago
  My girlfriend was using Stata. She's an epidemiologist.
asnyder 18y ago
Is it just me, or was this article not very well written? I felt it was all over the place. It brings up S, then mentions that S isn't open source. It mentions open source and brings up things like apache, the web, and Microsoft, I don't see how it relates much to R. Though, I'm probably spoiled due to the usual quality of news I get here.
earl 18y ago
The power of R (speaking as a very heavy user who has deployed it in multiple production environments and been using it for 5 years) is that it makes it very fast, easy, and natural to do statistics. It also has the nicest data structure I've ever seen for manipulating table data, called a data frame -- I'll elaborate, if anybody cares. In addition, it encourages people to create packages to extend the functionality. There are extant packages to do almost every analysis you can think of -- time series, kmeans, other clustering techniques, cox-box style analyses, regular maximum likelihood style GLM, hierarchical regression, HB, etc. Further, the amount of knowledge and the open source nature of the language, base, and packages encourage additional development and widespread adoption. See: http://cran.r-project.org/ http://cran.r-project.org/ and http://cran.r-project.org/web/views/ http://cran.r-project.org/web/views/ ^ is task views. Explore it -- it's well worth your time. The downsides are, well, it's slow for large data sets and debugging can be difficult. But as a desktop / rapid development platform for statistics it is without peer, IMO. ps -- unlike Matlab, which often costs thousands of dollars, and the Statistics Toolbox, more thousands, R is free. This is pretty important on its own -- instead of $5k per server and workstation and home pc, install it on any linux, Mac, or windows box you have and get to work for $0.00.
Prrometheus 18y ago
Could someone explain how this differs from Matlab, which is the most popular language for statistics and machine learning at my university?
- aposteriori 18y ago
  As earl said, it's free! But that's not the most important aspect (although nice for a student). It's an actual programming language. Like Python, you can use both non-OOP and OOP styles. You can define your own packages and namespaces (unlike MATLAB where there's just one big namespace). There are hundreds of contributed packages, you don't need to buy separate toolboxes. Not to forget, it's GPL, you can look at the source and learn a few things from people who know what they're doing (to name a few: Brian Ripley, Terry Therneau, Douglas Bates, Bill Venables). On the other hand, it can be a bit of a steep learning curve at the beginning, but I feel it's definitely worth it. It's not perfect, I stub my toe on obscure language features from time to time, but to paraphrase Winston Churchill, R is the worst form of statistical languages except all those other forms that have been tried from time to time... I think that Andrew Robinson's introduction is pretty decent (http://cran.r-project.org/doc/contrib/Robinson-icebreaker.pdf http://cran.r-project.org/doc/contrib/Robinson-icebreaker.pd...), but there are many others at http://cran.r-project.org/other-docs.html http://cran.r-project.org/other-docs.html
waldrews 18y ago
At least it's a functional language, and you can do things like manipulating code symbolically, showing its lisp heritage. The tooling, library integration, and debuggers aren't as good as, say, Python, though.
rdixit 18y ago
My 2 cents: Numpy+Scipy+Matplotlib and other packages, which u can download together in a convenient package at Enthought. That enthought distribution also comes with Ipython, which, is, REALLY nice. I checked out R, Sage and am still sometimes forced into Matlab, but u just can't beat a programming language (Python) which can be used OUTSIDE of whtever problem space you happen to be working on.
rdixit 18y ago
My 2 cents: Numpy+Scipy+Matplotlib and other packages, which u can download together in a convenient package at Enthought. That enthought distribution also comes with Ipython, which, is, REALLY nice. I checked out R, Sage and am still sometimes forced into Matlab, but u just can't beat a programming language (Python) which can be used OUTSIDE of whtever problem space you happen to be working on.
mojonixon 18y ago
"But R has also quickly found a following because statisticians, engineers and scientists without computer programming skills find it easy to use." whaaaa?

82 points by kalvin 18y ago

Meanwhile the creator of R wants to return to a Lisp-based statistics environment: http://books.google.com/books?id=8Cf16JkKz30C&pg=PA21&lpg=PA21 http://books.google.com/books?id=8Cf16JkKz30C&pg=PA21&#3...

jderick 18y ago
Another interesting paper hidden behind the academic firewall.
- Anon84 18y ago
  At least in the physics world people just post everything to arXiv and their personal websites. This way you only need to pay attention to a couple of places to keep up to date.
  - brent 18y ago
    Even in certain areas of computer science (specifically, machine learning) nearly all recent articles are open. I find in most cases that both statisticians and computer scientists tend to post links to pdfs or ps on their websites as well.
  - scott_s 18y ago
    As I said in another thread, most computer science papers are on authors' webpages. Then this one comes along - but it looks like the authors are stat people. I don't know what their culture is - and what kind of copyright agreements they sign. I can access the paper because my school subscribes. If anyone wants to read it, figure out how to email me and I'll send a copy. While searching, I did find a brief email from Ihaka talking about the jump to Lisp: https://stat.ethz.ch/pipermail/r-devel/2008-May/049501.html https://stat.ethz.ch/pipermail/r-devel/2008-May/049501.html
    - gaius 18y ago
      Thanks :-)
steveblgh 18y ago
Lush is a LISP-style environment for scientific computing: http://lush.sourceforge.net/ http://lush.sourceforge.net/
mwexler 18y ago
Statistics and Lisp have a long, long history together. Before R took off, many academic statisticians were using Lisp-Stat, Luke Tierney's tool from UMN. Luke has now been spending time helping R, and it shows in how it is starting to adopt more of a Lisp mentality. The referenced paper also shows the Lisp background from R (and S) history. R syntax can be confusing enough; Lisp really has a learning effort such that most analysts just don't have the time to invest in it unless they were trained in it early on.
twopoint718 18y ago
Maxima (http://maxima.sourceforge.net/ http://maxima.sourceforge.net/) is a nice CAS, sort of similar to Maple, IIRC. It is written in Lisp but hides that from the interface layer.

scott_s 18y ago

I am baffled that a programming language is getting coverage in the NYT. I'd like to figure out what, exactly, it is about this one that merits mainstream coverage, but I'm afraid I've been at the office too long and I think my perspective on the matter is permanently skewed. As an aside, I used R briefly for a econometrics/philosophy course I took. I recognized immediately it was a powerful, functional language. What I wonder, though, is if the scientific programming libraries in Python might eventually be a better environment. Surely there must be some R users here who can comment.

jderick 18y ago
Economics is hot right now?
- daniel-cussen 18y ago
  In my opinion it is. I have anticipated this downturn for the rich historical perspective on depressions it will offer.
Anon84 18y ago
if the scientific programming libraries in Python might eventually be a better environment. Well... there is R.py http://rpy.sourceforge.net/ http://rpy.sourceforge.net/ to give you most (if not all) the functionality of R in conjunction with all the scipy goodness.
showerst 18y ago
Having only very minimal experience with R, I get the impression that while python is fantastic for programmers, and the tools are probably close to or better than R's (no idea about library support), it's still a programming language, and requires one to think and structure things like a programmer. R, at least in the beginning, comes across more like a powerful set of Excel formulas, so I think a non-programmer might pick it up faster without having good programming form. (Not to say one is more powerful than the other, or that R programmers aren't as good as Python programmers, etc, I just think it's a difference in community viewpoints)

tokenadult 18y ago

"The co-creators of R express satisfaction that such companies profit from the fruits of their labor and that of hundreds of volunteers." That's an interesting reaction from the first designers of the program.

zandorg 18y ago

"We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet." - Ugh, SAS isn't about science, it's about administration.

bkj123 18y ago
What do you mean by "SAS isn't about science, it's about administration"
- zandorg 18y ago
  I meant SAP, but that or SAS is about 'business intelligence', which means running an organisation - staffing, payroll, etc, but certainly not jet engines. My Aunt programs in SAP for a University. So SAP determines how businesses are run, and are paid for the privelege.
  - neilc 18y ago
    SAP and SAS are completely different. SAS is for science (well, for statistical programming in general); SAS has nothing to do with SAP, which is for back-office stuff. The article and the quote are about statistics packages like SAS.
michaelneale 18y ago
What an idiot. There would be lots of freeware involved if not on the aircraft. And that is preferable to the alternative in many cases.
- aneesh 18y ago
  Well, for starters, the video screens on the seatbacks of 777s (and other aircraft) are running some flavor of Linux. But that's not a counterexample to her argument, per se, since the video screens aren't critical to the plane's operation.
  - 18y ago

wesm 18y ago

having programmed R on the job for some heavy statistics, I will say this: good for quick analyses but burdened by legacy functionality from the S+ days. I switched to Python/NumPy and rewrote all the R code I had, could not be happier with the results. Of course, you have to create your own data structures if you want something like R's data frame, but at least you have a rich language to do that with. however, if you need to do anything systematic, do NOT use R, bugs are elusive and extremely tedious to debug

yters 18y ago
I have used R for a number of minor projects. I really like its functional programming design, but its syntax can be obtuse. It's preferable to MatLab, IMHO. What do you consider to be the legacy aspects that weigh it down? What do you like better about NumPy? While I'm asking questions, have you tried Sage? If so, what do you think of it as a meta package of mathematical software? Also, here are interesting links I've found comparing R and similar statistical programs. Sorry, don't know much about Numpy's capabilities. R equivalent in speed to Matlab: http://www.sciviews.org/benchmark/index.html http://www.sciviews.org/benchmark/index.html Someone's research for a good data analysis language: http://www.cs.ubc.ca/~murphyk/Software/which_language.html http://www.cs.ubc.ca/~murphyk/Software/which_language.html
- wesm 18y ago
  One of R's big benefits is the huge amount of statistical functionality, for example numerous different quantilization algorithms. If you are working with a lot of heterogeneous data in R it becomes a real headache. Merging data frames seems like it should work like you think it should but if one of your sets of keys (strings) are 'factors' (what I am calling 'legacy S+ functionality', I'm sure they're useful for many algorithms), you'll end up with garbage. There's a hack you can put in your code ('options(stringsAsFactors=FALSE)') which alleviates some of this but in general aligning data I found to be a huge pain. If you're running regressions this is pretty important haven't tried sage but have heard good things. NumPy is a good alternative because it's extremely well implemented and has consistent behavior across the board. Extensibility (with Fortran, Cython/Pyrex, C/C++) is clean and easy. Never thought I'd write Fortran 77 code being born quite a few years after '77 but it's an easy way to speed up simple procedural algorithms 50x or more.

jessep 18y ago

Interesting. My girlfriend is a statistician for the WHO and they definitely still use SAS, at least in her area (calculating global burden of disease). I'm going to ask her if anyone there uses R.

jessep 18y ago
she says apparently there's a big debate going on at the WHO about whether everything should be done in R. currently they use R, Stata, and SAS.

manny 18y ago

I can't believe nobody here has mentioned PDL, the Perl Data Language: http://pdl.perl.org http://pdl.perl.org Admittedly, i think R and PDL do different things... (I have never played with R).

draegtun 18y ago
Looks like PDL == NumPy (http://news.ycombinator.com/item?id=363159 http://news.ycombinator.com/item?id=363159) There are CPAN modules to directly use R from Perl (for eg... R::* & Statistics::R). I have a stats friend who's been singing the praises of R & PDL for donkey years. /I3az/

bbgm 18y ago

In the life science space, R dominates research informatics. A large chunk of molecular profiling methods and techniques use R, or quite often the Bioconductor package, http://www.bioconductor.org/ http://www.bioconductor.org/ (from Gentleman's group). Most commercial bioinformatics apps also implement a number of methods using R and provide ability to implement R-based classifiers, etc. In the clinical space, it's all SAS. Pretty much the de facto standard.

stcredzero 18y ago
My girlfriend was using Stata. She's an epidemiologist.

asnyder 18y ago

Is it just me, or was this article not very well written? I felt it was all over the place. It brings up S, then mentions that S isn't open source. It mentions open source and brings up things like apache, the web, and Microsoft, I don't see how it relates much to R. Though, I'm probably spoiled due to the usual quality of news I get here.

earl 18y ago

The power of R (speaking as a very heavy user who has deployed it in multiple production environments and been using it for 5 years) is that it makes it very fast, easy, and natural to do statistics. It also has the nicest data structure I've ever seen for manipulating table data, called a data frame -- I'll elaborate, if anybody cares. In addition, it encourages people to create packages to extend the functionality. There are extant packages to do almost every analysis you can think of -- time series, kmeans, other clustering techniques, cox-box style analyses, regular maximum likelihood style GLM, hierarchical regression, HB, etc. Further, the amount of knowledge and the open source nature of the language, base, and packages encourage additional development and widespread adoption. See: http://cran.r-project.org/ http://cran.r-project.org/ and http://cran.r-project.org/web/views/ http://cran.r-project.org/web/views/ ^ is task views. Explore it -- it's well worth your time. The downsides are, well, it's slow for large data sets and debugging can be difficult. But as a desktop / rapid development platform for statistics it is without peer, IMO. ps -- unlike Matlab, which often costs thousands of dollars, and the Statistics Toolbox, more thousands, R is free. This is pretty important on its own -- instead of $5k per server and workstation and home pc, install it on any linux, Mac, or windows box you have and get to work for $0.00.

Prrometheus 18y ago

Could someone explain how this differs from Matlab, which is the most popular language for statistics and machine learning at my university?

aposteriori 18y ago
As earl said, it's free! But that's not the most important aspect (although nice for a student). It's an actual programming language. Like Python, you can use both non-OOP and OOP styles. You can define your own packages and namespaces (unlike MATLAB where there's just one big namespace). There are hundreds of contributed packages, you don't need to buy separate toolboxes. Not to forget, it's GPL, you can look at the source and learn a few things from people who know what they're doing (to name a few: Brian Ripley, Terry Therneau, Douglas Bates, Bill Venables). On the other hand, it can be a bit of a steep learning curve at the beginning, but I feel it's definitely worth it. It's not perfect, I stub my toe on obscure language features from time to time, but to paraphrase Winston Churchill, R is the worst form of statistical languages except all those other forms that have been tried from time to time... I think that Andrew Robinson's introduction is pretty decent (http://cran.r-project.org/doc/contrib/Robinson-icebreaker.pdf http://cran.r-project.org/doc/contrib/Robinson-icebreaker.pd...), but there are many others at http://cran.r-project.org/other-docs.html http://cran.r-project.org/other-docs.html

waldrews 18y ago

At least it's a functional language, and you can do things like manipulating code symbolically, showing its lisp heritage. The tooling, library integration, and debuggers aren't as good as, say, Python, though.

rdixit 18y ago

My 2 cents: Numpy+Scipy+Matplotlib and other packages, which u can download together in a convenient package at Enthought. That enthought distribution also comes with Ipython, which, is, REALLY nice. I checked out R, Sage and am still sometimes forced into Matlab, but u just can't beat a programming language (Python) which can be used OUTSIDE of whtever problem space you happen to be working on.

rdixit 18y ago

mojonixon 18y ago

"But R has also quickly found a following because statisticians, engineers and scientists without computer programming skills find it easy to use." whaaaa?