4 ms·

> includes The Intel Math Kernel Libraries, which bring multi-threaded computations to R. Say no more, I'm sold! Now that I can use multiple threads, please l

> includes The Intel Math Kernel Libraries, which bring multi-threaded computations to R. Say no more, I'm sold! Now that I can use multiple threads, please let me distribute my calculations over multiple machines and I can drop python. I used to be the compiler developer for a proprietary language that was vectorized like Matlab and R. If R was where it is today in 2000, we probably wouldn't have created a new language(Quicscript) for quants to use. I can't recommend R enough to anyone doing modelling. It's almost replaced excel for me and, in finance, that's pretty high praise :)

snoman 12y ago
I was looking into them yesterday, and the best source of info on what exactly REvolution R seems to be is from Wikipedia (Their sites is fraught with buzzwords and little info). That is, it's an R distro that has multi-threaded computation as you noticed, but also has the facility to deal with datasets larger than can be stored in memory and supposedly also has some integration with Hadoop.
- mbq 12y ago
  To be fair, free R+CRAN is also capable of all that stuff, not to mention that it can be easily compiled with ICC using MKL (provided that one has a license for them, but most academias have).
  - snoman 12y ago
    Oh certainly. I'm not biased one way or another, as I'm pretty ignorant of what's available. I've just been getting into the R space myself in the last few months in what free time I have available after work. My understanding of MRAN is that it's also something of a 'checkpoint' system for R package releases, so that you can distribute your r scripts with the version # of the packages used so that if anyone else runs them, they'll use the same packages and get the same results. It sounds much like the 'dll hell' of VB/C++ in the 90s.
jorgemarsal 12y ago
DistributedR is multicore and distributed. It may be what you are looking for. http://www.vertica.com/distributedr/ http://www.vertica.com/distributedr/ https://github.com/vertica/DistributedR https://github.com/vertica/DistributedR
etrain 12y ago
Check out SparkR - http://amplab-extras.github.io/SparkR-pkg/ http://amplab-extras.github.io/SparkR-pkg/
nkurz 12y ago
This doesn't negate what you are saying, but I happened to be helping a friend yesterday optimize her already optimized R routine by writing a C++ extension. In this case, a straightforward translation gave us about a 10x speedup, due much less memory shuffling. Switching to intersecting 2KB bitmaps with AVX2 intrinsics instead of non-vectorizable IntegerArrays (average length about 20 ints) gave another ~10x speedup. Profiling with 'perf', staring at the generated assembly, and choosing very specific syntax that g++ didn't mangle gave another ~2x (although Intel's icpc is still about 20% faster, and I haven't yet been able to trick g++ into generating the comparable). Was this worth it? In this case, I think so. It me took a day and half, and brings her per model runtime down from about 3 minutes to 1.5 seconds. The project requires running about 30,000 models, so the improvement will reduce her total CPU time down from about 2 months to about 12 hours. The AVX2 requirement means she can no longer run it on the usual cluster, but the speedup is enough that she do multiple runs per day on a standard Haswell desktop. This is to say, while there are great things about prototyping with R, and multithreading can't hurt, raw speed and efficiency may still be issues for some users.
- random_number 12y ago
  Cool result. But: "Was this worth it?" - looking at the last step of optimisation, it looks like that would be the hardest to do correctly without bugs, hardest to maintain, document and so on, but it produced a gain of only 12 hours of compute time (so cheap it's almost free). On an 8-5 schedule, 12 hours is 24. Was it really worth, or just fun?
  - nkurz 12y ago
    > Was it really worth, or just fun? That's the question for just about everything, isn't it? This was unpaid to help with her PhD thesis. Maybe that makes it "not worth it" from the start? Does the lack of financial incentive make "for fun" the only applicable answer? The project models the correlation between air quality in school classrooms and student absence rates. Maybe twice as fast means that she can do twice as many runs in the same amount of time, and produce results that are accurate to one decimal place more? She'd probably graduate in either case, and the chances that someone will act differently based on the difference in the results is very low. For that matter, the chance that someone will act at all based on results published in a PhD thesis is very low. Had someone been paying me by the hour to optimize for them, perhaps that would make it worth it to me but not to them? Is it just for fun if you are learning details and gaining experience that you can apply elsewhere? At what point does further learning become superfluous? I've never figured these out. As it is, I learn more and more about optimizing code, my social capital among friends increases, and my financial state grows worse and worse.
pjmlp 12y ago
R was totally unknown to me until around three years ago. Nowadays I see it everywhere. Oracle is using R for big data analysis. And there is a version being ported to the JVM. And Julia is looking quite good. I see Python getting lots of heat if they don't sort out the issues with CPython.

snoman 12y ago

I was looking into them yesterday, and the best source of info on what exactly REvolution R seems to be is from Wikipedia (Their sites is fraught with buzzwords and little info). That is, it's an R distro that has multi-threaded computation as you noticed, but also has the facility to deal with datasets larger than can be stored in memory and supposedly also has some integration with Hadoop.

mbq 12y ago
To be fair, free R+CRAN is also capable of all that stuff, not to mention that it can be easily compiled with ICC using MKL (provided that one has a license for them, but most academias have).
- snoman 12y ago
  Oh certainly. I'm not biased one way or another, as I'm pretty ignorant of what's available. I've just been getting into the R space myself in the last few months in what free time I have available after work. My understanding of MRAN is that it's also something of a 'checkpoint' system for R package releases, so that you can distribute your r scripts with the version # of the packages used so that if anyone else runs them, they'll use the same packages and get the same results. It sounds much like the 'dll hell' of VB/C++ in the 90s.

jorgemarsal 12y ago

DistributedR is multicore and distributed. It may be what you are looking for. http://www.vertica.com/distributedr/ http://www.vertica.com/distributedr/ https://github.com/vertica/DistributedR https://github.com/vertica/DistributedR

etrain 12y ago

Check out SparkR - http://amplab-extras.github.io/SparkR-pkg/ http://amplab-extras.github.io/SparkR-pkg/

nkurz 12y ago

This doesn't negate what you are saying, but I happened to be helping a friend yesterday optimize her already optimized R routine by writing a C++ extension. In this case, a straightforward translation gave us about a 10x speedup, due much less memory shuffling. Switching to intersecting 2KB bitmaps with AVX2 intrinsics instead of non-vectorizable IntegerArrays (average length about 20 ints) gave another ~10x speedup. Profiling with 'perf', staring at the generated assembly, and choosing very specific syntax that g++ didn't mangle gave another ~2x (although Intel's icpc is still about 20% faster, and I haven't yet been able to trick g++ into generating the comparable). Was this worth it? In this case, I think so. It me took a day and half, and brings her per model runtime down from about 3 minutes to 1.5 seconds. The project requires running about 30,000 models, so the improvement will reduce her total CPU time down from about 2 months to about 12 hours. The AVX2 requirement means she can no longer run it on the usual cluster, but the speedup is enough that she do multiple runs per day on a standard Haswell desktop. This is to say, while there are great things about prototyping with R, and multithreading can't hurt, raw speed and efficiency may still be issues for some users.

random_number 12y ago
Cool result. But: "Was this worth it?" - looking at the last step of optimisation, it looks like that would be the hardest to do correctly without bugs, hardest to maintain, document and so on, but it produced a gain of only 12 hours of compute time (so cheap it's almost free). On an 8-5 schedule, 12 hours is 24. Was it really worth, or just fun?
- nkurz 12y ago
  > Was it really worth, or just fun? That's the question for just about everything, isn't it? This was unpaid to help with her PhD thesis. Maybe that makes it "not worth it" from the start? Does the lack of financial incentive make "for fun" the only applicable answer? The project models the correlation between air quality in school classrooms and student absence rates. Maybe twice as fast means that she can do twice as many runs in the same amount of time, and produce results that are accurate to one decimal place more? She'd probably graduate in either case, and the chances that someone will act differently based on the difference in the results is very low. For that matter, the chance that someone will act at all based on results published in a PhD thesis is very low. Had someone been paying me by the hour to optimize for them, perhaps that would make it worth it to me but not to them? Is it just for fun if you are learning details and gaining experience that you can apply elsewhere? At what point does further learning become superfluous? I've never figured these out. As it is, I learn more and more about optimizing code, my social capital among friends increases, and my financial state grows worse and worse.

pjmlp 12y ago

R was totally unknown to me until around three years ago. Nowadays I see it everywhere. Oracle is using R for big data analysis. And there is a version being ported to the JVM. And Julia is looking quite good. I see Python getting lots of heat if they don't sort out the issues with CPython.