7 ms·

A Future for R: A Comprehensive Overview

michaelsbradley 10y ago
Also see this slide deck, by the author: http://www.aroma-project.org/share/presentations/BengtssonH_20160628-useR2016/BengtssonH_20160628-A_Future_for_R,useR2016.html http://www.aroma-project.org/share/presentations/BengtssonH_...
deleted 10y ago
[deleted]
johnmyleswhite 10y ago
I'd like to see the title changed to something like "Futures for R" since it's so tongue-in-cheek that it's effectively clickbait.
- projectramo 10y ago
  That might sound like trading commodity futures.
nosound_warmup 10y ago
This feels a little like using a hammer for cutting down a tree. You could do it, but there really are better tools for that particular job.
- pvaldes 10y ago
  Could you please explain in more detail?
- rcthompson 10y ago
  I think there are perfect use cases for this style of code. For instance, I sometimes find myself writing code something like this: some_value <- compute_some_value(dataset1) other_value <- compute_other_value(dataset2) final_answer <- some_value + other_value where compute_some_value() and compute_other_value() are independent and both of them take a long time to run, so they would benefit from running in parallel. However, actually running them in parallel is tricky, because most parallel interfaces in R are modelled after lapply, running a single function on multiple elements of a list, and this doesn't fit that mold. You could parallelize it manually using primitives such as parallel::mcparallel, and delayedAssign, but you don't get error handling/propagation, and your code gets super messy with the implementation details of your parallelization strategy. And if you do parallelize it and then someone else calls your code in parallel, now you get too many parallel processes and risk running of memory and ending up in swapping hell. The bottom line is that code such as the above generally just doesn't get parallelized, because the only way of doing so (as far as I know) requires pointing several guns at your foot. So this package looks very interesting and useful to me, and I also think it provides a good set of primitives with which to implement yet another "multi-backend parallel lapply" package with advantages over the others, such as doing its best to ensure consistent behavior across the different "backends". (Edit: Also see jonchang's comment along similar lines.)
  - michaelsbradley 10y ago
    See also doFuture[1], an adaptor for the foreach[2] package. [1] https://cran.r-project.org/web/packages/doFuture/vignettes/doFuture.html https://cran.r-project.org/web/packages/doFuture/vignettes/d... [2] https://cran.r-project.org/web/packages/foreach/index.html https://cran.r-project.org/web/packages/foreach/index.html
  - baldeagle 10y ago
    Thank you for explaining this; I was trying to see how this would be useful. Could this be used to do parallel data load (like read from csv and database at the same time?)
    - huac 10y ago
      yes, you could draw from multiple data sources at the same time. csv_data %<-% read.csv("myfile.csv") tsv_data %<-% read.tsv("otherfile.tsv") csv_data %>% left_join(tsv_data, by = "id_user")
jonchang 10y ago
So I've actually tried to use the futures package. While it's very clean for certain types of tasks, there are a few problems that I think are inherent to the way R deals with its parallel packages (which the futures package is built on top of). Futures is great for tasks where you have some kind of task workflow like: # slow task 1 --------, # ----> task 3 # slow task 2 --------' do_stuff <- function(input1, input2) { result1 <- slow_task1(input1) result2 <- slow_task2(input2) task3(result1, result2) } Because then you can just do something like: library(futures) plan(multicore) do_stuff <- function(input1, input2) { result1 %<-% slow_task1(input1) result2 %<-% slow_task2(input2) task3(result1, result2) } And boom, you can have two tasks running in parallel and everything "just works." It's extremely nice to use thanks to R's promises capability. Where it falls down is when you try to load up a bunch of futures at once... I'm not clear on the implementation details, but from what I can tell every parallel task is assigned a "port" on your system, but if there is a port conflict (or the OS doesn't "release" (?) the port quickly enough) tasks simply die with an inscrutable error. I've found that it's necessary to 1. ensure that only one "set" of parallel tasks are running at one time, and 2. create a central "port registry" and manually assign ports randomly within nonoverlapping ranges for parallel tasks. It's straightforward but frustrating to do. Finally (and I don't know if the futures package has updated since I tried it out last year) it doesn't work on Windows, which is a problem for many R users.
- apathy 10y ago
  File issues with Henrik. He actually reads them.
  - jonchang 10y ago
    These are not issues with the futures package. These are issues with the way parallelism is implemented in R.
    - apathy 10y ago
      Yes and no. Multicore/multisession is not the only option for parallelism in R. The default approach is to use fork() on *nix and... hell, I don't use Windows, so I don't really care how it works there. But there are other options. Depending on your use case it may be fixable. What topology are you using where this is breaking down, and could the synchronization/communication be queued with something like rredis or rrqueue? All of this assuming you want to use R for your work, as opposed to an external dependency injection framework (or some crap like Torque).
apathy 10y ago
See also https://mitpress.mit.edu/sicp/full-text/sicp/book/node70.html https://mitpress.mit.edu/sicp/full-text/sicp/book/node70.htm... for fun background reading. Or if you have an actual CS degree, review from freshman year.
- protomyth 10y ago
  > Or if you have an actual CS degree, review from freshman year. You do realize that actual CS degrees happened before 1985?
  - apathy 10y ago
    Before 2007, you mean? Because that's when SICP was deprecated for MIT 6.001. Happily, it's still available: http://web.mit.edu/alexmv/6.037/ http://web.mit.edu/alexmv/6.037/
    - protomyth 10y ago
      I was going for the original date of publication but I see your point.
haddr 10y ago
This definitely a great feature! Now, with this in hand can we have for instance a multithreaded (or in some other parallel way) web server or even REST API for R? Talking from the practical perspective, the biggest problem with wide adoption of R is the problem of integration. Sometimes you just want to have a single module in R, and the rest of the system in some other technology. I know there are ways to do it, but not without quite high technical debt. On the other hand having native microservice-like integration could probably help. I know we are close, but not sure how close.
- kirillseva 10y ago
  This is not REST API per se, but it's something that we've used in production for a couple years now and it's working like a charm https://github.com/robertzk/microserver https://github.com/robertzk/microserver https://github.com/kirillseva/rain https://github.com/kirillseva/rain This way our rails app can communicate with R prediction backend and we end up using the right tool for each task
  - haddr 10y ago
    Thanks! Just a question: is this microserver blocking? I mean when a request takes 2-3 sec, can it server other requests in parallel?
    - kirillseva 10y ago
      It's a wrapper around https://github.com/rstudio/httpuv https://github.com/rstudio/httpuv I'm pretty sure it's blocking though, R doesn't really have good concurrency primitives. However, you can launch multiple R processes behind nginx and be able to serve N requests concurrently. Running this in an autoscaling group on AWS provides enough resilience in practice
      - haddr 10y ago
        Yes, R is missing those, but there is some workarounds, like Shiny server (enterprise version) and OpenCPU for instance.
rcthompson 10y ago
I actually hacked together an almost functional system similar to this a few years ago. It used the same primitives, delayedAssign and parallel::mcparallel, to implement parallel-evaluated lazy promises. It was nearly useful, but I couldn't get it to work when passing one promised value to another lazy expression, presumably because only the process that forked a subprocess can read the value it returns, so the second forked process can't evaluate the promose. It looks like this package solves that problem by forcing evaluation of any implicit futures before passing them to another future. I'm definitely interested in trying this out.
insulanian 10y ago
Can someone explain in few sentences (or point me to the place where I can get the answer) what is the benefit of using R over some functional language like Haskel, OCaml or F#? Which use-cases does R shine in over those languages? Is the popularity driven by R the language or its libraries?
- mziel 10y ago
  While obviously the answer is libraries and data types (DataFrame is a very powerful semantic, replicated later in Python, Julia, Scala/Spark), R definitely IS a functional language: - closures, lambdas - high-order functions - a lot R processing is map/fold-like (even though it's lapply instead of map)
  - michaelsbradley 10y ago
    There's also the `Map`, `Filter`, and `Reduce` functions provided by the base library, i.e. in addition to the `*apply` family of functions.
- blahi 10y ago
  Libraries. If you want to do statistics in any other language, you have to reimplement 80% of the diagnostics and tests by hand and a good portion of methods and algorithms (no, not just esoteric ones).
- bpyne 10y ago
  I took a course recently in ML that focused on Support Vector Machines (SVM), Decision Trees, and Neural Nets. The professor chose R because of the libraries. SVM's, Decision Trees, and Neural Nets are non-trivial to write libraries for. In order to produce similar libraries for any other language, you need either a good grounding in math (linear algebra, PDE's, and projective geometry at the least) or a well-developed ability to translate from one language - R source for instance - to the language of your choice without understanding completely what you're translating. Regardless of your background, it will take a significant amount of time and focus to accomplish it. Our professor mentioned that Python may have libraries available now. He likes Python in general. In his research, he uses R simply due to familiarity.

Also see this slide deck, by the author: http://www.aroma-project.org/share/presentations/BengtssonH_20160628-useR2016/BengtssonH_20160628-A_Future_for_R,useR2016.html http://www.aroma-project.org/share/presentations/BengtssonH_...

deleted 10y ago

[deleted]

johnmyleswhite 10y ago

I'd like to see the title changed to something like "Futures for R" since it's so tongue-in-cheek that it's effectively clickbait.

projectramo 10y ago
That might sound like trading commodity futures.

nosound_warmup 10y ago

This feels a little like using a hammer for cutting down a tree. You could do it, but there really are better tools for that particular job.

pvaldes 10y ago
Could you please explain in more detail?
rcthompson 10y ago
I think there are perfect use cases for this style of code. For instance, I sometimes find myself writing code something like this: some_value <- compute_some_value(dataset1) other_value <- compute_other_value(dataset2) final_answer <- some_value + other_value where compute_some_value() and compute_other_value() are independent and both of them take a long time to run, so they would benefit from running in parallel. However, actually running them in parallel is tricky, because most parallel interfaces in R are modelled after lapply, running a single function on multiple elements of a list, and this doesn't fit that mold. You could parallelize it manually using primitives such as parallel::mcparallel, and delayedAssign, but you don't get error handling/propagation, and your code gets super messy with the implementation details of your parallelization strategy. And if you do parallelize it and then someone else calls your code in parallel, now you get too many parallel processes and risk running of memory and ending up in swapping hell. The bottom line is that code such as the above generally just doesn't get parallelized, because the only way of doing so (as far as I know) requires pointing several guns at your foot. So this package looks very interesting and useful to me, and I also think it provides a good set of primitives with which to implement yet another "multi-backend parallel lapply" package with advantages over the others, such as doing its best to ensure consistent behavior across the different "backends". (Edit: Also see jonchang's comment along similar lines.)
- michaelsbradley 10y ago
  See also doFuture[1], an adaptor for the foreach[2] package. [1] https://cran.r-project.org/web/packages/doFuture/vignettes/doFuture.html https://cran.r-project.org/web/packages/doFuture/vignettes/d... [2] https://cran.r-project.org/web/packages/foreach/index.html https://cran.r-project.org/web/packages/foreach/index.html
- baldeagle 10y ago
  Thank you for explaining this; I was trying to see how this would be useful. Could this be used to do parallel data load (like read from csv and database at the same time?)
  - huac 10y ago
    yes, you could draw from multiple data sources at the same time. csv_data %<-% read.csv("myfile.csv") tsv_data %<-% read.tsv("otherfile.tsv") csv_data %>% left_join(tsv_data, by = "id_user")

jonchang 10y ago

So I've actually tried to use the futures package. While it's very clean for certain types of tasks, there are a few problems that I think are inherent to the way R deals with its parallel packages (which the futures package is built on top of). Futures is great for tasks where you have some kind of task workflow like: # slow task 1 --------, # ----> task 3 # slow task 2 --------' do_stuff <- function(input1, input2) { result1 <- slow_task1(input1) result2 <- slow_task2(input2) task3(result1, result2) } Because then you can just do something like: library(futures) plan(multicore) do_stuff <- function(input1, input2) { result1 %<-% slow_task1(input1) result2 %<-% slow_task2(input2) task3(result1, result2) } And boom, you can have two tasks running in parallel and everything "just works." It's extremely nice to use thanks to R's promises capability. Where it falls down is when you try to load up a bunch of futures at once... I'm not clear on the implementation details, but from what I can tell every parallel task is assigned a "port" on your system, but if there is a port conflict (or the OS doesn't "release" (?) the port quickly enough) tasks simply die with an inscrutable error. I've found that it's necessary to 1. ensure that only one "set" of parallel tasks are running at one time, and 2. create a central "port registry" and manually assign ports randomly within nonoverlapping ranges for parallel tasks. It's straightforward but frustrating to do. Finally (and I don't know if the futures package has updated since I tried it out last year) it doesn't work on Windows, which is a problem for many R users.

apathy 10y ago
File issues with Henrik. He actually reads them.
- jonchang 10y ago
  These are not issues with the futures package. These are issues with the way parallelism is implemented in R.
  - apathy 10y ago
    Yes and no. Multicore/multisession is not the only option for parallelism in R. The default approach is to use fork() on *nix and... hell, I don't use Windows, so I don't really care how it works there. But there are other options. Depending on your use case it may be fixable. What topology are you using where this is breaking down, and could the synchronization/communication be queued with something like rredis or rrqueue? All of this assuming you want to use R for your work, as opposed to an external dependency injection framework (or some crap like Torque).

apathy 10y ago

See also https://mitpress.mit.edu/sicp/full-text/sicp/book/node70.html https://mitpress.mit.edu/sicp/full-text/sicp/book/node70.htm... for fun background reading. Or if you have an actual CS degree, review from freshman year.

protomyth 10y ago
> Or if you have an actual CS degree, review from freshman year. You do realize that actual CS degrees happened before 1985?
- apathy 10y ago
  Before 2007, you mean? Because that's when SICP was deprecated for MIT 6.001. Happily, it's still available: http://web.mit.edu/alexmv/6.037/ http://web.mit.edu/alexmv/6.037/
  - protomyth 10y ago
    I was going for the original date of publication but I see your point.

haddr 10y ago

This definitely a great feature! Now, with this in hand can we have for instance a multithreaded (or in some other parallel way) web server or even REST API for R? Talking from the practical perspective, the biggest problem with wide adoption of R is the problem of integration. Sometimes you just want to have a single module in R, and the rest of the system in some other technology. I know there are ways to do it, but not without quite high technical debt. On the other hand having native microservice-like integration could probably help. I know we are close, but not sure how close.

kirillseva 10y ago
This is not REST API per se, but it's something that we've used in production for a couple years now and it's working like a charm https://github.com/robertzk/microserver https://github.com/robertzk/microserver https://github.com/kirillseva/rain https://github.com/kirillseva/rain This way our rails app can communicate with R prediction backend and we end up using the right tool for each task
- haddr 10y ago
  Thanks! Just a question: is this microserver blocking? I mean when a request takes 2-3 sec, can it server other requests in parallel?
  - kirillseva 10y ago
    It's a wrapper around https://github.com/rstudio/httpuv https://github.com/rstudio/httpuv I'm pretty sure it's blocking though, R doesn't really have good concurrency primitives. However, you can launch multiple R processes behind nginx and be able to serve N requests concurrently. Running this in an autoscaling group on AWS provides enough resilience in practice
    - haddr 10y ago
      Yes, R is missing those, but there is some workarounds, like Shiny server (enterprise version) and OpenCPU for instance.

rcthompson 10y ago

I actually hacked together an almost functional system similar to this a few years ago. It used the same primitives, delayedAssign and parallel::mcparallel, to implement parallel-evaluated lazy promises. It was nearly useful, but I couldn't get it to work when passing one promised value to another lazy expression, presumably because only the process that forked a subprocess can read the value it returns, so the second forked process can't evaluate the promose. It looks like this package solves that problem by forcing evaluation of any implicit futures before passing them to another future. I'm definitely interested in trying this out.

insulanian 10y ago

Can someone explain in few sentences (or point me to the place where I can get the answer) what is the benefit of using R over some functional language like Haskel, OCaml or F#? Which use-cases does R shine in over those languages? Is the popularity driven by R the language or its libraries?

mziel 10y ago
While obviously the answer is libraries and data types (DataFrame is a very powerful semantic, replicated later in Python, Julia, Scala/Spark), R definitely IS a functional language: - closures, lambdas - high-order functions - a lot R processing is map/fold-like (even though it's lapply instead of map)
- michaelsbradley 10y ago
  There's also the `Map`, `Filter`, and `Reduce` functions provided by the base library, i.e. in addition to the `*apply` family of functions.
blahi 10y ago
Libraries. If you want to do statistics in any other language, you have to reimplement 80% of the diagnostics and tests by hand and a good portion of methods and algorithms (no, not just esoteric ones).
bpyne 10y ago
I took a course recently in ML that focused on Support Vector Machines (SVM), Decision Trees, and Neural Nets. The professor chose R because of the libraries. SVM's, Decision Trees, and Neural Nets are non-trivial to write libraries for. In order to produce similar libraries for any other language, you need either a good grounding in math (linear algebra, PDE's, and projective geometry at the least) or a well-developed ability to translate from one language - R source for instance - to the language of your choice without understanding completely what you're translating. Regardless of your background, it will take a significant amount of time and focus to accomplish it. Our professor mentioned that Python may have libraries available now. He likes Python in general. In his research, he uses R simply due to familiarity.