3 ms·
The CPU power would not be too hard to utilize, if one's task is amenable to fairly coarse-grained parallelization, which is rather easy to implement in R scrip
by michaelsbradley 10y ago
The CPU power would not be too hard to utilize, if one's task is amenable to fairly coarse-grained parallelization, which is rather easy to implement in R scripts with the help of foreach[1] and doFuture[2].
Such an approach worked to great effect for me recently, when I needed to perform a `zoo::rollapply`[3] across a time series with tens of millions of rows. The speedup when throwing more cores (7, on my laptop) at it is roughly linear. If I ever need to scale up the analysis to hundreds of millions of rows, the 128 vCPUs of an x1 EC2 instance would be well worth the $$/hour.
[1] https://cran.r-project.org/web/packages/foreach/index.html https://cran.r-project.org/web/packages/foreach/index.html
[2] https://cran.r-project.org/web/packages/doFuture/index.html https://cran.r-project.org/web/packages/doFuture/index.html
[3] http://www.rdocumentation.org/topics/1505094 http://www.rdocumentation.org/topics/1505094
- huac 10y agoShameless self promotion: my `slide_apply` function shows 5-10x improvement over `zoo::rollapply` (despite no Rcpp call, somehow...) https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee https://gist.github.com/stillmatic/fadfd3269b900e1fd7ee if your function has a well-defined rolling form, e.g. rolling mean or standard deviation, you should use a pre-optimized function for that. package `catools` has a bunch of useful ones.
- michaelsbradley 10y agoThanks! As it happens, my function doesn't conform to any pre-optimized facilities, though I wish it did. At some point, I may look into writing an optimized variation, with the help of Rcpp, or even using the (inspiring!) fortran approach of quantmod/xts.