3 ms·

You can rent an r3.8xlarge instance with 344GB RAM on AWS for $2.66/hr. If that isn't enough, an x1.32xlarge instance with ~2TB RAM is $13.34/hr. Assuming your

You can rent an r3.8xlarge instance with 344GB RAM on AWS for $2.66/hr. If that isn't enough, an x1.32xlarge instance with ~2TB RAM is $13.34/hr. Assuming your data is already in AWS, of course. Of course those machines have a crap ton of CPU power (I'm explicitly not saying cores because I'm mystified by what vCPU really means) as well which you will be hard pressed to take advantage of but if it's RAM you need they're super-cheap. If you prefer Azure, a G5 instance with 448GB RAM is available for $9.65/hr. Not a lot of need for Spark/SparkR for the vast majority of data sets given the cheapness of compute these days.

qaq 10y ago
vCPU is a hyperthread
michaelsbradley 10y ago
The CPU power would not be too hard to utilize, if one's task is amenable to fairly coarse-grained parallelization, which is rather easy to implement in R scripts with the help of foreach[1] and doFuture[2]. Such an approach worked to great effect for me recently, when I needed to perform a `zoo::rollapply`[3] across a time series with tens of millions of rows. The speedup when throwing more cores (7, on my laptop) at it is roughly linear. If I ever need to scale up the analysis to hundreds of millions of rows, the 128 vCPUs of an x1 EC2 instance would be well worth the $$/hour. [1] https://cran.r-project.org/web/packages/foreach/index.html https://cran.r-project.org/web/packages/foreach/index.html [2] https://cran.r-project.org/web/packages/doFuture/index.html https://cran.r-project.org/web/packages/doFuture/index.html [3] http://www.rdocumentation.org/topics/1505094 http://www.rdocumentation.org/topics/1505094
- huac 10y ago
  Shameless self promotion: my `slide_apply` function shows 5-10x improvement over `zoo::rollapply` (despite no Rcpp call, somehow...) https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee if your function has a well-defined rolling form, e.g. rolling mean or standard deviation, you should use a pre-optimized function for that. package `catools` has a bunch of useful ones.
  - michaelsbradley 10y ago
    Thanks! As it happens, my function doesn't conform to any pre-optimized facilities, though I wish it did. At some point, I may look into writing an optimized variation, with the help of Rcpp, or even using the (inspiring!) fortran approach of quantmod/xts.

qaq 10y ago

vCPU is a hyperthread

michaelsbradley 10y ago

The CPU power would not be too hard to utilize, if one's task is amenable to fairly coarse-grained parallelization, which is rather easy to implement in R scripts with the help of foreach[1] and doFuture[2]. Such an approach worked to great effect for me recently, when I needed to perform a `zoo::rollapply`[3] across a time series with tens of millions of rows. The speedup when throwing more cores (7, on my laptop) at it is roughly linear. If I ever need to scale up the analysis to hundreds of millions of rows, the 128 vCPUs of an x1 EC2 instance would be well worth the $$/hour. [1] https://cran.r-project.org/web/packages/foreach/index.html https://cran.r-project.org/web/packages/foreach/index.html [2] https://cran.r-project.org/web/packages/doFuture/index.html https://cran.r-project.org/web/packages/doFuture/index.html [3] http://www.rdocumentation.org/topics/1505094 http://www.rdocumentation.org/topics/1505094

huac 10y ago
Shameless self promotion: my `slide_apply` function shows 5-10x improvement over `zoo::rollapply` (despite no Rcpp call, somehow...) https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee if your function has a well-defined rolling form, e.g. rolling mean or standard deviation, you should use a pre-optimized function for that. package `catools` has a bunch of useful ones.
- michaelsbradley 10y ago
  Thanks! As it happens, my function doesn't conform to any pre-optimized facilities, though I wish it did. At some point, I may look into writing an optimized variation, with the help of Rcpp, or even using the (inspiring!) fortran approach of quantmod/xts.