10 ms·

The future of R - pessimistic thoughts by R founder Ross Ihaka

76 points by TalGalili 16y ago

The recent post on the shortcomings of R has attracted a huge number of readers and Ross Ihaka has now posted a detailed comment that is fairly pessimistic… Given the directions drafted in this comment from the father of R (along with Robert Gentleman), I once again re-post this comment as a main entry to advertise more broadly its contents. (Obviously, the whole debate is now far beyond my reach!)

chrismealy 16y ago
I would welcome a replacement for R just because the new language's name might be easier to google. I hope they don't call it Q.
- nanairo 16y ago
  Yes, please! It seems nowadays one of the most important feature in a name is to be unique from the point of view of Google. :) (incidentally, if you are a mpaa operative, call your blockbusters with just one very generic name... a lot easier than suing half the world, and just as annoying ;) )
- frankc 16y ago
  There is actually rseek.org just for this purpose.
- T_S_ 16y ago
  The top five links for "R" are all the right ones. Google seems to have you covered.
- makmanalp 16y ago
  google: r statistics foo
- TalGalili 16y ago
  BTW, that was my motivation for starting the www.r-bloggers.com website. It now has over 110 bloggers (who write about R) there. When I started and looked for them on google, all I could find was bloggers who wrote about pirates :D
  - SkyMarshal 16y ago
    Google thought R = Arrrgghh? Lol...
- macromicro 16y ago
  Q already exists: http://en.wikipedia.org/wiki/Q_(programming_language) http://en.wikipedia.org/wiki/Q_(programming_language)
gwern 16y ago
It almost sounds like they want to switch to Haskell.
- ghotli 16y ago
  Or perhaps incanter with all it's lispy goodness. "Incanter is a Clojure-based, R-like platform for statistical computing and graphics." http://incanter.org/ http://incanter.org/
  - gwern 16y ago
    Wouldn't get you pervasive laziness, though I don't know whether R is lazy by default or like Clojure in that you have to ask for it.
    - zaphar 16y ago
      if you use Clojures primitives you pretty much end up with laziness by default. for, map and most of the other list processing functions all return lazy lists. So while Clojure does technically require you to "turn on" laziness in most cases you'll find it's turned on already.
    - dododo 16y ago
      in R, arguments are passed around lazily, akin to scala's =>. otherwise, evaluation is not lazy (as in Haskell) unless done so explicitly (as in force/delay for scheme). i think essentially no one using R professionally would accept Haskell or Clojure as a substitute. having spoken to a few: lisp-like syntax is "unreadable", and Haskell is too much effort.
  - muuh-gnu 16y ago
    Ihaka himself seems to be eyeing with Common Lisp as the next best statistical environment: http://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-2008.pdf http://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-2...
    - 16y ago
nanairo 16y ago
To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my this news... if there's gonna be a need for R to have its own language, speed seems to be the most important distinguishing feature. A bit like Fortran is still used in science. (incidentally... don't let people tell you otherwise: Fortran(90+) is a very nice language... much more pleasurable than C to use and gives you better performance (unless you know a lot about compilers and compiling flags... but most scientist don't ^_^))
- ewjordan 16y ago
  But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. Indeed - it would be a shame for them to start over from scratch and end up coming up with a brand new language, brand new syntax, brand new quirks, brand new performance problems, etc., while they could have simply searched around a bit for something that's already mature and somewhat optimized as well as suiting their needs. If they wanted to add on to or modify an existing language (for instance, to provide more concise syntax for some of the things that are more important in statistics than in general purpose programming), that would be just fine, it could become a dialect of some other language, but starting fresh seems like an awful waste of energy... Something that ran on the JVM would be awesome, they'd have no trouble at all rebuilding the massive library of contributions.
  - equark 16y ago
    Fear not, some of us are working on this very hard.
    - knowtheory 16y ago
      o? what is it that you're doing?
adbge 16y ago
> The license will need to a better job of protecting work donated to the commons than GPL2 seems to have done. I’m not willing to have any more of my work purloined by the likes of Revolution Analytics, so I’ll be looking for better protection from the license (and being a lot more careful about who I work with). Not having used R or being more than passingly familiar with it, I'm wondering if anyone could shed some light on what this is about? I notice that on the Revolution Analytics website, they are selling an "enterprise" version of R which they claim has a number of advantages over the mainstream form of R. How exactly is a proprietary form of R even able to exist if R's codebase is GPL'd?
- naner 16y ago
  It appears they release the sources for the GPL parts and their changes to the GPL parts: http://www.revolutionanalytics.com/downloads/gpl-sources.php http://www.revolutionanalytics.com/downloads/gpl-sources.php And from their FAQ: Yes. CRAN R is licensed under the GNU General Public License (version 2). As permitted under that license, Revolution R is based on a modified version of CRAN R. The source code to that modified version (including a list of changes) is available for download when you download the binary version of Revolution R, as required by the GPL. As an open-source company, Revolution Analytics respects the Free and Open Source Software philosophy and respects all free-software licenses. Revolution Analytics is also a direct supporter of the open-source R project: financially (as a benefactor of the R Foundation), organizationally (by sponsoring events and user group meetings), and technically (by contributing or modifications to CRAN R back to the community).
  - _delirium 16y ago
    Hmm, that seems to be interpreting the GPL as closer to what the LGPL's intent is. I assume they're able to do this because their other tools (e.g. IDE) are careful not to actually link to R, but only call it via things like shared files and pipes?
- aamar 16y ago
  IANAL. This is discussed at http://www.gnu.org/licenses/gpl-faq.html#GPLInProprietarySystem http://www.gnu.org/licenses/gpl-faq.html#GPLInProprietarySys.... RA claims to have adopted this "arm's length" approach, releasing their direct extensions to the community. They ask me for identifying information to let me download the software; don't know if that's against the rules. Critically, their enterprise product is pitched as a collection of separate tools: IDE, debugger, analysis tool, etc. There are a few other ways that people have made GPL'd code proprietary: (1) providing a hosted service instead of distributing software, or (2) distributing only a hardware appliance that contains inextricable software. The latter is controversial with respect to GPL2; GPL3 more clearly prohibits it.
haberman 16y ago
To me the problem with R isn't performance problems, which I've never run into myself, but rather the complicated and confusing semantics of its data types. R's aggregate data types are: vector, matrix, array, dataframe, and list. The semantics of these types and the relationships between them are extremely confusing. I wish I had gathered examples of this so I could be more specific, but I have basically come to the conclusion that I will never get familiar enough with them to do any better than random guessing until it works right. And I've written somewhat in-depth analyses in R.
- nanairo 16y ago
  I may be wrong but I think that: list => are basically hash, or an array that can have mixed objects inside vector, matrix, array => are all the same thing. They are what in most computer languages are called arrays, and can have only one type. The difference between those three is just the number of dimensions (vector:1, matrix:2, array:3+). dataframe I will concede is a little more complex, and I still have some problem with it. But I basically think of it as a table, where a row represents a value (say temperature) and the column different measuremnts. So, for example: rows=> temperature, humidity, hours of light, peak UV columns=> Day1, day2, day3, day4, ... Hope that helps.
  - pashields 16y ago
    Lists are hashes on acid. The default return value of indexing in is NULL. That's a weird semantic for a list, but maybe not so much a hash. The weird part is that if you explicitly assign null to an element of the list, it deletes the element. That's particularly weird because the list has knows all the elements in it, so it's not like it can't tell the difference between a value that has been set to NULL and a value that has never been set. See http://gist.github.com/578110 http://gist.github.com/578110 for a little transcript.
  - hadley 16y ago
    Vectors are not the same as 1d matrices/arrays because a vector does not have direction (i.e. it's not a row vector or a column vector, but will work as either depending on circumstance). > all.equal(1:10, matrix(1:10, ncol = 1)) [1] "Attributes: < target is NULL, current is list >" [2] "target is numeric, current is matrix" > all.equal(matrix(1:10, ncol = 1), array(1:10, c(10, 1))) [1] TRUE
pajarito 16y ago
a translator from R to Maxima would be another idea. Maxima is Lisp based, use an algol like language, is used in education and is free. But developer are scare so more hands are needed. Maxima can use BLAS and other libraries for numeric computation, and it support many implementations of Lisp (sbcl, gcl, cll, ecl ...), some of their member are working in a java based lisp (Arms beast?). With so many tools available it is a something to consider.
hogu 16y ago
python has, RPy2 so you can still access R's statistics libraries http://scikits.appspot.com/statsmodels http://scikits.appspot.com/statsmodels, and http://pandas.sourceforge.net/ http://pandas.sourceforge.net/ (dataframes) plus, scipy has a decent stats library as well (random variables, etc) still rough around the edges, but a good solution in my opinion

76 points by TalGalili 16y ago

chrismealy 16y ago

I would welcome a replacement for R just because the new language's name might be easier to google. I hope they don't call it Q.

nanairo 16y ago
Yes, please! It seems nowadays one of the most important feature in a name is to be unique from the point of view of Google. :) (incidentally, if you are a mpaa operative, call your blockbusters with just one very generic name... a lot easier than suing half the world, and just as annoying ;) )
frankc 16y ago
There is actually rseek.org just for this purpose.
T_S_ 16y ago
The top five links for "R" are all the right ones. Google seems to have you covered.
makmanalp 16y ago
google: r statistics foo
TalGalili 16y ago
BTW, that was my motivation for starting the www.r-bloggers.com website. It now has over 110 bloggers (who write about R) there. When I started and looked for them on google, all I could find was bloggers who wrote about pirates :D
- SkyMarshal 16y ago
  Google thought R = Arrrgghh? Lol...
macromicro 16y ago
Q already exists: http://en.wikipedia.org/wiki/Q_(programming_language) http://en.wikipedia.org/wiki/Q_(programming_language)

gwern 16y ago

It almost sounds like they want to switch to Haskell.

ghotli 16y ago
Or perhaps incanter with all it's lispy goodness. "Incanter is a Clojure-based, R-like platform for statistical computing and graphics." http://incanter.org/ http://incanter.org/
- gwern 16y ago
  Wouldn't get you pervasive laziness, though I don't know whether R is lazy by default or like Clojure in that you have to ask for it.
  - zaphar 16y ago
    if you use Clojures primitives you pretty much end up with laziness by default. for, map and most of the other list processing functions all return lazy lists. So while Clojure does technically require you to "turn on" laziness in most cases you'll find it's turned on already.
  - dododo 16y ago
    in R, arguments are passed around lazily, akin to scala's =>. otherwise, evaluation is not lazy (as in Haskell) unless done so explicitly (as in force/delay for scheme). i think essentially no one using R professionally would accept Haskell or Clojure as a substitute. having spoken to a few: lisp-like syntax is "unreadable", and Haskell is too much effort.
- muuh-gnu 16y ago
  Ihaka himself seems to be eyeing with Common Lisp as the next best statistical environment: http://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-2008.pdf http://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-2...
  - 16y ago

nanairo 16y ago

To be honest I've been using R a bit lately for my work and while I like it I don't find it at all innovative. That's not a criticism of R: the libraries it has are amazing, as well as the mindshare among people who care of statistics. But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. So I am kind of pleased my this news... if there's gonna be a need for R to have its own language, speed seems to be the most important distinguishing feature. A bit like Fortran is still used in science. (incidentally... don't let people tell you otherwise: Fortran(90+) is a very nice language... much more pleasurable than C to use and gives you better performance (unless you know a lot about compilers and compiling flags... but most scientist don't ^_^))

ewjordan 16y ago
But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. Indeed - it would be a shame for them to start over from scratch and end up coming up with a brand new language, brand new syntax, brand new quirks, brand new performance problems, etc., while they could have simply searched around a bit for something that's already mature and somewhat optimized as well as suiting their needs. If they wanted to add on to or modify an existing language (for instance, to provide more concise syntax for some of the things that are more important in statistics than in general purpose programming), that would be just fine, it could become a dialect of some other language, but starting fresh seems like an awful waste of energy... Something that ran on the JVM would be awesome, they'd have no trouble at all rebuilding the massive library of contributions.
- equark 16y ago
  Fear not, some of us are working on this very hard.
  - knowtheory 16y ago
    o? what is it that you're doing?

adbge 16y ago

> The license will need to a better job of protecting work donated to the commons than GPL2 seems to have done. I’m not willing to have any more of my work purloined by the likes of Revolution Analytics, so I’ll be looking for better protection from the license (and being a lot more careful about who I work with). Not having used R or being more than passingly familiar with it, I'm wondering if anyone could shed some light on what this is about? I notice that on the Revolution Analytics website, they are selling an "enterprise" version of R which they claim has a number of advantages over the mainstream form of R. How exactly is a proprietary form of R even able to exist if R's codebase is GPL'd?

naner 16y ago
It appears they release the sources for the GPL parts and their changes to the GPL parts: http://www.revolutionanalytics.com/downloads/gpl-sources.php http://www.revolutionanalytics.com/downloads/gpl-sources.php And from their FAQ: Yes. CRAN R is licensed under the GNU General Public License (version 2). As permitted under that license, Revolution R is based on a modified version of CRAN R. The source code to that modified version (including a list of changes) is available for download when you download the binary version of Revolution R, as required by the GPL. As an open-source company, Revolution Analytics respects the Free and Open Source Software philosophy and respects all free-software licenses. Revolution Analytics is also a direct supporter of the open-source R project: financially (as a benefactor of the R Foundation), organizationally (by sponsoring events and user group meetings), and technically (by contributing or modifications to CRAN R back to the community).
- _delirium 16y ago
  Hmm, that seems to be interpreting the GPL as closer to what the LGPL's intent is. I assume they're able to do this because their other tools (e.g. IDE) are careful not to actually link to R, but only call it via things like shared files and pipes?
aamar 16y ago
IANAL. This is discussed at http://www.gnu.org/licenses/gpl-faq.html#GPLInProprietarySystem http://www.gnu.org/licenses/gpl-faq.html#GPLInProprietarySys.... RA claims to have adopted this "arm's length" approach, releasing their direct extensions to the community. They ask me for identifying information to let me download the software; don't know if that's against the rules. Critically, their enterprise product is pitched as a collection of separate tools: IDE, debugger, analysis tool, etc. There are a few other ways that people have made GPL'd code proprietary: (1) providing a hosted service instead of distributing software, or (2) distributing only a hardware appliance that contains inextricable software. The latter is controversial with respect to GPL2; GPL3 more clearly prohibits it.

haberman 16y ago

To me the problem with R isn't performance problems, which I've never run into myself, but rather the complicated and confusing semantics of its data types. R's aggregate data types are: vector, matrix, array, dataframe, and list. The semantics of these types and the relationships between them are extremely confusing. I wish I had gathered examples of this so I could be more specific, but I have basically come to the conclusion that I will never get familiar enough with them to do any better than random guessing until it works right. And I've written somewhat in-depth analyses in R.

nanairo 16y ago
I may be wrong but I think that: list => are basically hash, or an array that can have mixed objects inside vector, matrix, array => are all the same thing. They are what in most computer languages are called arrays, and can have only one type. The difference between those three is just the number of dimensions (vector:1, matrix:2, array:3+). dataframe I will concede is a little more complex, and I still have some problem with it. But I basically think of it as a table, where a row represents a value (say temperature) and the column different measuremnts. So, for example: rows=> temperature, humidity, hours of light, peak UV columns=> Day1, day2, day3, day4, ... Hope that helps.
- pashields 16y ago
  Lists are hashes on acid. The default return value of indexing in is NULL. That's a weird semantic for a list, but maybe not so much a hash. The weird part is that if you explicitly assign null to an element of the list, it deletes the element. That's particularly weird because the list has knows all the elements in it, so it's not like it can't tell the difference between a value that has been set to NULL and a value that has never been set. See http://gist.github.com/578110 http://gist.github.com/578110 for a little transcript.
- hadley 16y ago
  Vectors are not the same as 1d matrices/arrays because a vector does not have direction (i.e. it's not a row vector or a column vector, but will work as either depending on circumstance). > all.equal(1:10, matrix(1:10, ncol = 1)) [1] "Attributes: < target is NULL, current is list >" [2] "target is numeric, current is matrix" > all.equal(matrix(1:10, ncol = 1), array(1:10, c(10, 1))) [1] TRUE

pajarito 16y ago

a translator from R to Maxima would be another idea. Maxima is Lisp based, use an algol like language, is used in education and is free. But developer are scare so more hands are needed. Maxima can use BLAS and other libraries for numeric computation, and it support many implementations of Lisp (sbcl, gcl, cll, ecl ...), some of their member are working in a java based lisp (Arms beast?). With so many tools available it is a something to consider.

hogu 16y ago

python has, RPy2 so you can still access R's statistics libraries http://scikits.appspot.com/statsmodels http://scikits.appspot.com/statsmodels, and http://pandas.sourceforge.net/ http://pandas.sourceforge.net/ (dataframes) plus, scipy has a decent stats library as well (random variables, etc) still rough around the edges, but a good solution in my opinion