5 ms·

But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. Inde

But I wonder why R actually needs to exist as its own language. It seems it could be recast in Ruby for example or one of the latest functional languages. Indeed - it would be a shame for them to start over from scratch and end up coming up with a brand new language, brand new syntax, brand new quirks, brand new performance problems, etc., while they could have simply searched around a bit for something that's already mature and somewhat optimized as well as suiting their needs. If they wanted to add on to or modify an existing language (for instance, to provide more concise syntax for some of the things that are more important in statistics than in general purpose programming), that would be just fine, it could become a dialect of some other language, but starting fresh seems like an awful waste of energy... Something that ran on the JVM would be awesome, they'd have no trouble at all rebuilding the massive library of contributions.

equark 16y ago
Fear not, some of us are working on this very hard.
- knowtheory 16y ago
  o? what is it that you're doing?
  - equark 16y ago
    Building a next generation of statistical computing.
    - phren0logy 16y ago
      Would you be willing to be more specific? That's not very helpful.
      - equark 16y ago
        I can't be yet, unfortunately. But there are definitely several different groups of very talented people working on this problem (think MIT and Harvard PhDs). I suspect some very interesting work will come out in the next year or two. Whether they are open source remains another question.
        filobloomz 16y ago
        Austin Heap? Is that you?
    - knowtheory 16y ago
      I was hoping you'd be at least the slightest bit more specific :) are you working in a proprietary startup? are you working in academia? presumably you're working on a codebase with the intent to distribute it to users at some point?
deleted 16y ago
[deleted]
bigfudge 16y ago
Perhaps I'm wrong, but scipy and numpy seem to do a lot of what they want, and if they started there a lot more of their effort could go into porting the libraries.
- earl 16y ago
  Yes, but (to my shock), their speed makes even R look fast. Which I didn't think as possible. My problem was taking a matrix market formatted matrix, loading it, turning it into a sparse column vector representation, then computing norms of the columns. It was running for roughly 8 hours on 12MM columns. R of all things was running faster. I reimplemented in java and it takes < 1 second.
  - studer 16y ago
    Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather odd. I assume you checked for memory leaks and swapping. Did you do any profiling?
    - cdavid 16y ago
      To be fair, the sparse matrix package is still very rough. I am highly skeptical of the 1sec in java vs 8 hours in scipy, though. In general, numpy/scipy is quite faster than R: it does not have the pass by value semantics for once. I am also skeptical about writing a "new" R: the main value of R is in the R packages. Any new language would threw that away.
      - earl 16y ago
        cdavid -- see http://news.ycombinator.com/item?id=1688694 http://news.ycombinator.com/item?id=1688694
Quiark 16y ago
For JVM, there is Incanter which is a statistics library written in Clojure. It is backed by Parallel Colt for the heavy number lifting. Note that I'm not trying to say that Clojure would be a scientist-friendly language :)
- nanairo 16y ago
  How does Incanter work on HPC? R is pretty awful from that point of view, and if there ever will be room for a specialised statistical language it's got to be able to do massive number crunching. I've heard bad things of JVM for tightly coupled jobs on HPC (though I know there's been some improvement: e.g. a lot of work done by EPCC in Edinburgh). Does Clojure manages to offer a good parallel implementation on top of the JVM or has no work been done in this area?
  - Quiark 16y ago
    According to the website, Parallel Colt supports multicore machines. There's no mention of MPI though. As far as I know, Incanter only wraps it, so it does not influence the performance that much..
- piggybox 16y ago
  A lot of people blame Lisp for too many brackets, but I generally feel they are not the issue after reading Lisp for a while. What may make it scientist-unfriendly is to write math formulas in infix grammar.
  - regularfry 16y ago
    Incanter has the $= macro for writing infix expressions, so I don't think that need be an issue.

equark 16y ago

Fear not, some of us are working on this very hard.

knowtheory 16y ago
o? what is it that you're doing?
- equark 16y ago
  Building a next generation of statistical computing.
  - phren0logy 16y ago
    Would you be willing to be more specific? That's not very helpful.
    - equark 16y ago
      I can't be yet, unfortunately. But there are definitely several different groups of very talented people working on this problem (think MIT and Harvard PhDs). I suspect some very interesting work will come out in the next year or two. Whether they are open source remains another question.
      - filobloomz 16y ago
        Austin Heap? Is that you?
  - knowtheory 16y ago
    I was hoping you'd be at least the slightest bit more specific :) are you working in a proprietary startup? are you working in academia? presumably you're working on a codebase with the intent to distribute it to users at some point?

deleted 16y ago

[deleted]

bigfudge 16y ago

Perhaps I'm wrong, but scipy and numpy seem to do a lot of what they want, and if they started there a lot more of their effort could go into porting the libraries.

earl 16y ago
Yes, but (to my shock), their speed makes even R look fast. Which I didn't think as possible. My problem was taking a matrix market formatted matrix, loading it, turning it into a sparse column vector representation, then computing norms of the columns. It was running for roughly 8 hours on 12MM columns. R of all things was running faster. I reimplemented in java and it takes < 1 second.
- studer 16y ago
  Given that numpy/scipy is basically a collection of C/C++/Fortran primitives, chances are that you managed to write a program that spent very little time actually computing things, and a lot of time doing something else. Not sure what that could be, though; even low-level algorithms usually run 10-100x slower than C speed if naively coded in plain Python, so a 30000x slowdown using a specialized library sounds rather odd. I assume you checked for memory leaks and swapping. Did you do any profiling?
  - cdavid 16y ago
    To be fair, the sparse matrix package is still very rough. I am highly skeptical of the 1sec in java vs 8 hours in scipy, though. In general, numpy/scipy is quite faster than R: it does not have the pass by value semantics for once. I am also skeptical about writing a "new" R: the main value of R is in the R packages. Any new language would threw that away.
    - earl 16y ago
      cdavid -- see http://news.ycombinator.com/item?id=1688694 http://news.ycombinator.com/item?id=1688694

Quiark 16y ago

For JVM, there is Incanter which is a statistics library written in Clojure. It is backed by Parallel Colt for the heavy number lifting. Note that I'm not trying to say that Clojure would be a scientist-friendly language :)

nanairo 16y ago
How does Incanter work on HPC? R is pretty awful from that point of view, and if there ever will be room for a specialised statistical language it's got to be able to do massive number crunching. I've heard bad things of JVM for tightly coupled jobs on HPC (though I know there's been some improvement: e.g. a lot of work done by EPCC in Edinburgh). Does Clojure manages to offer a good parallel implementation on top of the JVM or has no work been done in this area?
- Quiark 16y ago
  According to the website, Parallel Colt supports multicore machines. There's no mention of MPI though. As far as I know, Incanter only wraps it, so it does not influence the performance that much..
piggybox 16y ago
A lot of people blame Lisp for too many brackets, but I generally feel they are not the issue after reading Lisp for a while. What may make it scientist-unfriendly is to write math formulas in infix grammar.
- regularfry 16y ago
  Incanter has the $= macro for writing infix expressions, so I don't think that need be an issue.