7 ms·

R at Microsoft

106 points by vladiim 11y ago

IndianAstronaut 11y ago
The write once run anywhere concept is intriguing. R has major scaling issues and a lot of the algorithms and packages in the CRAN just don't scale to any reasonable amounts of data. It really was designed in an era of controlled experiments with minimal data.
- mjfl 11y ago
  R is a funny language because there are a lot of things in it that are just objectively bad, yet I still love it. Probably because of the community and CRAN. Of course, R is really just some syntax wrapped over C++, and it is very extendable using C and C++ when needed, so I don't think you can really say, in absolute, that R doesn't scale. Someone has written a package to scale whatever problem in R, you just need to find it.
- stdbrouw 11y ago
  > an era of controlled experiments with minimal data This world hasn't disappeared, y'know. Many of the specialized packages on CRAN deal with things like missing data, removing selection bias from longitudinal studies and distinguishing mediated from direct effects. These concerns are by and large not even applicable to the data the average data scientist deals with, so why even bother making them scalable? On the ML front, on the other hand, R has never really been a forerunner.
  - blumkvist 11y ago
    I don't know why this comment is downvoted. It is true. The vast majority of businesses will never deal with machine learning. ML has very limited applications in the "real world", as opposed to regression analysis with some small samples.
    - ska 11y ago
      Your point that simple regressions with small(ish) sample sets are important is valid. This: > "The vast majority of businesses will never deal with machine learning." However, is myopic. It plausibly will not hold for much longer in most areas, and is already not true for a significant number of industries.
      - blumkvist 11y ago
        I find ML and "3d printing" very similar in the expectations and extent business will adopt them.
        ska 11y ago
        On what basis? They are not similar in any respect that I can think of. 3D printing has had limited impact for prototyping in the many decades it has been around, and in the expected quarters. That's pretty much what every analysis I've seen except a very few boosters have predicted, for what it's worth. On the other hand ML has fundamentally changed: document processing, search engines, remote sensing, retail sales, e-commerce, etc. ; It has also strongly impacted areas of logistics and shipping, manufacturing, (e.g. computer vision & robotics), medicine, law enforcement, finance, defense, also etc. It's also poised to find market advantages in many non-obvious industries as larger scale data becomes available... unclear how far that will extend. Where's the parallel you see?
        IndianAstronaut 11y ago
        Many of the companies I have interviewed and all the ones I have worked for have set up ML and analytics teams in the last 5 years. It is a growing area and people are starting to understand its utility.
        blumkvist 11y ago
        Poorly understood, high hopes, a lot lower practical applications. Predictions are not as applicable as one might think at first glance. I hope you do not confuse machine learning with statistics and data mining. They are very different.
- nl 11y ago
  The work going on in Spark[1] is pretty interesting from this point of view. Using R functions via closures against Spark RDDs in a cluster has the potential to deliver pretty good performance against large scale data. [1] https://amplab-extras.github.io/SparkR-pkg/ https://amplab-extras.github.io/SparkR-pkg/
  - IndianAstronaut 11y ago
    I am very excited about the SparkR package. I have it set up on my desktop box and look forward to seeing what models and learning algorithms will work well with it. There really is no runner up to R in terms of the variety of statistical models, now what it needs is to scale up.
jordigh 11y ago
Does someone know how does this work with regards to the GPL? I have heard that they have a complete reimplementation of R. Really? What about all of the GPL'ed packages at CRAN, do they just not ship those? I also know that the R developers don't really enforce the GPL. Is that what's going on here?
- jlarocco 11y ago
  According to [1] R "will be run inside a sandbox process within SQL Server itself". If that's correct then there's not much to it. MS includes a link somewhere saying "Go here for the R source code." As far as CRAN goes, the licenses of the packages there have nothing to do with the license of the R implementations they run on. Even if MS has their own implementation of R, it should be able to download and run packages from CRAN without MS having to worry about it. [1] http://blog.revolutionanalytics.com/2015/05/r-in-sql-server.html http://blog.revolutionanalytics.com/2015/05/r-in-sql-server....
  - DannyBee 11y ago
    " If that's correct then there's not much to it." The FSF would beg to differ the last time i looked, depending on the situation: http://www.gnu.org/licenses/gpl-faq.en.html#GPLPluginsInNF http://www.gnu.org/licenses/gpl-faq.en.html#GPLPluginsInNF is the closest to their position on this. Basically, if you can function call into R from SQL/etc, and it's just not just a subprocess completely independent, their view is that you'd have to GPL the main process.
    - tormeh 11y ago
      That's a very strict position to take. So using a GPL'd library in my code means my code is now GPL as well? I understand now why many companies blanket ban GPL software.
      - jordigh 11y ago
        It's not very strict; it's the raison d'être of the GPL. The LGPL was written precisely to allow libraries without copyleft. A big chunk of CRAN is library-based and depends on a binary interface to R, and is GPL'ed. The FSF also does not think that whether the GPL'ed program is a separate process or not determines whether the bundle as a whole is GPL'ed or not. In their opinion, if you bundle the whole thing and it acts as a whole, and the communication between the GPL'ed components and the non-free parts are "intimate enough"[1], then the whole bundle is copylefted. It seems to me that bundling R with SQL server and tightly coupling the two, even across different processes, could well qualify. Of course, if the R developers do not want to enforce their copyleft, none of this matters. [1] https://www.gnu.org/licenses/gpl-faq.html#MereAggregation https://www.gnu.org/licenses/gpl-faq.html#MereAggregation
david_seah 11y ago
Anyone able to read the slides? I have difficulty accessing the slides.
- facorreia 11y ago
  They worked for me (Chrome, Linux). Perhaps you can try the direct link: http://www.slideshare.net/RevolutionAnalytics/r-at-microsoft?ref=http://blog.revolutionanalytics.com/2015/06/r-at-microsoft.html http://www.slideshare.net/RevolutionAnalytics/r-at-microsoft...
blumkvist 11y ago
I have to say I am very disappointed in the information, after the title got me so excited. No announcements about Excel? I really hope R lands in Excel, it will open big opportunities for developers and businesses. Also there was a guy from MS, who created python tools for VS and asked if anyone would want R tools for VS. After receiving a very warm response, never heard from him again. What's up with that? Parallel to that RA has an IDE, which uses the VS shell. Will we have more streamlined R workflow in VS? This is important for a lot of people... What about R.NET?
- Ntrails 11y ago
  If R lands in excel we can see even more proliferation of unstable excel "applications" with unmaintainable VBA, hard (if not impossible) to follow logic all wrapped up in a program where the next release might flat out break everything. Unless microsoft address some of the core issues with using excel as a "tools" platform - I'd prefer they didn't encourage it any more than they already do.
  - blumkvist 11y ago
    You're exaggerating, for sure.
    - Ntrails 11y ago
      I'm probably underselling it actually. Since excel 2013 came out several spreadsheets which used to function in 2010 no longer do, and anything over ~50mb has become significantly less stable. I've spent literally days debugging excel tools with VBA split all around between sheets, modules, classes etc with no documentation. Not to mention that additional logic lives in the various formulae inside worksheets (which may or may not be dynamically updated multiple times during code execution and then trigger a worksheet_changed macro). The value of source control is pretty limited for an excel sheet, and I've been in a situation whereby a source controlled sheet could no longer be saved without excel crashing. The end solution was to copy all of the modules/sheets/extraneous code into a new workbook. A highly manual process. It appeared to work - but good luck proving I didn't screw it up. Links to other workbooks are significantly more unstable in 2013 (they're probably a terrible idea anyway, but the company considers it the best way to affirm the data being used for calculations is right). Excel 2013 launches differently to 2010 breaking every excel tool which relied on launch args. I can probably find a bunch of other reasons that excel is something that literally gives me nightmares. I'm not exaggerating, and based on a few conversations with people in various finance companies this is far from isolated to myself and my company.
jokoon 11y ago
I don't know a lot about R, but apart from the fact that it's made for stats, isn't it a great language because it's also aimed to handle embarrassingly parallel programs, and no other other language do it as well as R ? I guess R is used in machine learning fields. Is R opencl enabled, or is it planned to be ?
- JohnDoe365 11y ago
  Julia http://julialang.org/ http://julialang.org/ Btw. almost down-vote for "embarrassingly"
  - jokoon 11y ago
    https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel You mean I misspelled ?
- capnrefsmmat 11y ago
  R doesn't have built-in parallelism features that I know of. There are a lot of packages, though: http://cran.r-project.org/web/views/HighPerformanceComputing.html http://cran.r-project.org/web/views/HighPerformanceComputing...
  - mziel 11y ago
    Most of the things can be transformed through lapply family (and d/plyr equivalents). d/plyr has simple parallel=TRUE. For other problems use mclapply(). That's basically all you need on a day-to-day basis.
  - grayclhn 11y ago
    The 'parallel' package has been included in recent versions of R and has RNGs for parallel execution and variations of the `apply` functions. For "embarrassingly parallel" calculations these work as drop-in replacements. It's built-in in the same way that MASS and lattice are.
- Lofkin 11y ago
trengrj 11y ago
Microsoft buying Revolution Analytics is another piece of evidence shoes Microsoft has its A-game back. R is massively popular in the statistical community, it is basically the PHP of analytics (huge standard library with no namespaces but very easy to get started with and powerful for advanced users). Additionally R is one of the few languages where you can be productive in a non-unix environment. At my old Windows workplace, R was the only open source language with working network NTLM authentication out of the box. Everything can be self-contained in RStudio, an interpreter and gui package manager to stop you jumping to the command line, and a help system to stop you using man. By buying becomming closely associated with R, Microsoft is more likely to be a component in large analytics investments i.e. "let's get Microsoft's R distro, oh and why not their big data plaform too?", and ensures its services and databases are well integrated in this upcoming language.
- fixxer 11y ago
  "PHP of analytics" I started laughing, then I realized you meant that as a compliment. R has a great community and a ton of unique packages, though many of those great packages are thin wrappers over Fortran/C libraries and are available in a host of other languages (Python, Julia) that, in my opinion, have better features when it comes to testing and group-based development. In my experience, R degenerates into a rat's nest more easily than the other candidates. I've been hired twice for this explicit reason. BTW, I think your comparison with PHP is apt. R has many of the same strengths and flaws.
  - taylorwc 11y ago
    Slight typo: compliment is correct here, rather than complement. Excellent discussion.
    - deleted 11y ago
      [deleted]

106 points by vladiim 11y ago

The write once run anywhere concept is intriguing. R has major scaling issues and a lot of the algorithms and packages in the CRAN just don't scale to any reasonable amounts of data. It really was designed in an era of controlled experiments with minimal data.

mjfl 11y ago
R is a funny language because there are a lot of things in it that are just objectively bad, yet I still love it. Probably because of the community and CRAN. Of course, R is really just some syntax wrapped over C++, and it is very extendable using C and C++ when needed, so I don't think you can really say, in absolute, that R doesn't scale. Someone has written a package to scale whatever problem in R, you just need to find it.
stdbrouw 11y ago
> an era of controlled experiments with minimal data This world hasn't disappeared, y'know. Many of the specialized packages on CRAN deal with things like missing data, removing selection bias from longitudinal studies and distinguishing mediated from direct effects. These concerns are by and large not even applicable to the data the average data scientist deals with, so why even bother making them scalable? On the ML front, on the other hand, R has never really been a forerunner.
- blumkvist 11y ago
  I don't know why this comment is downvoted. It is true. The vast majority of businesses will never deal with machine learning. ML has very limited applications in the "real world", as opposed to regression analysis with some small samples.
  - ska 11y ago
    Your point that simple regressions with small(ish) sample sets are important is valid. This: > "The vast majority of businesses will never deal with machine learning." However, is myopic. It plausibly will not hold for much longer in most areas, and is already not true for a significant number of industries.
    - blumkvist 11y ago
      I find ML and "3d printing" very similar in the expectations and extent business will adopt them.
      - ska 11y ago
        On what basis? They are not similar in any respect that I can think of. 3D printing has had limited impact for prototyping in the many decades it has been around, and in the expected quarters. That's pretty much what every analysis I've seen except a very few boosters have predicted, for what it's worth. On the other hand ML has fundamentally changed: document processing, search engines, remote sensing, retail sales, e-commerce, etc. ; It has also strongly impacted areas of logistics and shipping, manufacturing, (e.g. computer vision & robotics), medicine, law enforcement, finance, defense, also etc. It's also poised to find market advantages in many non-obvious industries as larger scale data becomes available... unclear how far that will extend. Where's the parallel you see?
        IndianAstronaut 11y ago
        Many of the companies I have interviewed and all the ones I have worked for have set up ML and analytics teams in the last 5 years. It is a growing area and people are starting to understand its utility.
        blumkvist 11y ago
        Poorly understood, high hopes, a lot lower practical applications. Predictions are not as applicable as one might think at first glance. I hope you do not confuse machine learning with statistics and data mining. They are very different.
nl 11y ago
The work going on in Spark[1] is pretty interesting from this point of view. Using R functions via closures against Spark RDDs in a cluster has the potential to deliver pretty good performance against large scale data. [1] https://amplab-extras.github.io/SparkR-pkg/ https://amplab-extras.github.io/SparkR-pkg/
- IndianAstronaut 11y ago
  I am very excited about the SparkR package. I have it set up on my desktop box and look forward to seeing what models and learning algorithms will work well with it. There really is no runner up to R in terms of the variety of statistical models, now what it needs is to scale up.

jordigh 11y ago

Does someone know how does this work with regards to the GPL? I have heard that they have a complete reimplementation of R. Really? What about all of the GPL'ed packages at CRAN, do they just not ship those? I also know that the R developers don't really enforce the GPL. Is that what's going on here?

jlarocco 11y ago
According to [1] R "will be run inside a sandbox process within SQL Server itself". If that's correct then there's not much to it. MS includes a link somewhere saying "Go here for the R source code." As far as CRAN goes, the licenses of the packages there have nothing to do with the license of the R implementations they run on. Even if MS has their own implementation of R, it should be able to download and run packages from CRAN without MS having to worry about it. [1] http://blog.revolutionanalytics.com/2015/05/r-in-sql-server.html http://blog.revolutionanalytics.com/2015/05/r-in-sql-server....
- DannyBee 11y ago
  " If that's correct then there's not much to it." The FSF would beg to differ the last time i looked, depending on the situation: http://www.gnu.org/licenses/gpl-faq.en.html#GPLPluginsInNF http://www.gnu.org/licenses/gpl-faq.en.html#GPLPluginsInNF is the closest to their position on this. Basically, if you can function call into R from SQL/etc, and it's just not just a subprocess completely independent, their view is that you'd have to GPL the main process.
  - tormeh 11y ago
    That's a very strict position to take. So using a GPL'd library in my code means my code is now GPL as well? I understand now why many companies blanket ban GPL software.
    - jordigh 11y ago
      It's not very strict; it's the raison d'être of the GPL. The LGPL was written precisely to allow libraries without copyleft. A big chunk of CRAN is library-based and depends on a binary interface to R, and is GPL'ed. The FSF also does not think that whether the GPL'ed program is a separate process or not determines whether the bundle as a whole is GPL'ed or not. In their opinion, if you bundle the whole thing and it acts as a whole, and the communication between the GPL'ed components and the non-free parts are "intimate enough"[1], then the whole bundle is copylefted. It seems to me that bundling R with SQL server and tightly coupling the two, even across different processes, could well qualify. Of course, if the R developers do not want to enforce their copyleft, none of this matters. [1] https://www.gnu.org/licenses/gpl-faq.html#MereAggregation https://www.gnu.org/licenses/gpl-faq.html#MereAggregation

david_seah 11y ago

Anyone able to read the slides? I have difficulty accessing the slides.

facorreia 11y ago
They worked for me (Chrome, Linux). Perhaps you can try the direct link: http://www.slideshare.net/RevolutionAnalytics/r-at-microsoft?ref=http://blog.revolutionanalytics.com/2015/06/r-at-microsoft.html http://www.slideshare.net/RevolutionAnalytics/r-at-microsoft...

blumkvist 11y ago

I have to say I am very disappointed in the information, after the title got me so excited. No announcements about Excel? I really hope R lands in Excel, it will open big opportunities for developers and businesses. Also there was a guy from MS, who created python tools for VS and asked if anyone would want R tools for VS. After receiving a very warm response, never heard from him again. What's up with that? Parallel to that RA has an IDE, which uses the VS shell. Will we have more streamlined R workflow in VS? This is important for a lot of people... What about R.NET?

Ntrails 11y ago
If R lands in excel we can see even more proliferation of unstable excel "applications" with unmaintainable VBA, hard (if not impossible) to follow logic all wrapped up in a program where the next release might flat out break everything. Unless microsoft address some of the core issues with using excel as a "tools" platform - I'd prefer they didn't encourage it any more than they already do.
- blumkvist 11y ago
  You're exaggerating, for sure.
  - Ntrails 11y ago
    I'm probably underselling it actually. Since excel 2013 came out several spreadsheets which used to function in 2010 no longer do, and anything over ~50mb has become significantly less stable. I've spent literally days debugging excel tools with VBA split all around between sheets, modules, classes etc with no documentation. Not to mention that additional logic lives in the various formulae inside worksheets (which may or may not be dynamically updated multiple times during code execution and then trigger a worksheet_changed macro). The value of source control is pretty limited for an excel sheet, and I've been in a situation whereby a source controlled sheet could no longer be saved without excel crashing. The end solution was to copy all of the modules/sheets/extraneous code into a new workbook. A highly manual process. It appeared to work - but good luck proving I didn't screw it up. Links to other workbooks are significantly more unstable in 2013 (they're probably a terrible idea anyway, but the company considers it the best way to affirm the data being used for calculations is right). Excel 2013 launches differently to 2010 breaking every excel tool which relied on launch args. I can probably find a bunch of other reasons that excel is something that literally gives me nightmares. I'm not exaggerating, and based on a few conversations with people in various finance companies this is far from isolated to myself and my company.

jokoon 11y ago

I don't know a lot about R, but apart from the fact that it's made for stats, isn't it a great language because it's also aimed to handle embarrassingly parallel programs, and no other other language do it as well as R ? I guess R is used in machine learning fields. Is R opencl enabled, or is it planned to be ?

JohnDoe365 11y ago
Julia http://julialang.org/ http://julialang.org/ Btw. almost down-vote for "embarrassingly"
- jokoon 11y ago
  https://en.wikipedia.org/wiki/Embarrassingly_parallel https://en.wikipedia.org/wiki/Embarrassingly_parallel You mean I misspelled ?
capnrefsmmat 11y ago
R doesn't have built-in parallelism features that I know of. There are a lot of packages, though: http://cran.r-project.org/web/views/HighPerformanceComputing.html http://cran.r-project.org/web/views/HighPerformanceComputing...
- mziel 11y ago
  Most of the things can be transformed through lapply family (and d/plyr equivalents). d/plyr has simple parallel=TRUE. For other problems use mclapply(). That's basically all you need on a day-to-day basis.
- grayclhn 11y ago
  The 'parallel' package has been included in recent versions of R and has RNGs for parallel execution and variations of the `apply` functions. For "embarrassingly parallel" calculations these work as drop-in replacements. It's built-in in the same way that MASS and lattice are.
Lofkin 11y ago

trengrj 11y ago

Microsoft buying Revolution Analytics is another piece of evidence shoes Microsoft has its A-game back. R is massively popular in the statistical community, it is basically the PHP of analytics (huge standard library with no namespaces but very easy to get started with and powerful for advanced users). Additionally R is one of the few languages where you can be productive in a non-unix environment. At my old Windows workplace, R was the only open source language with working network NTLM authentication out of the box. Everything can be self-contained in RStudio, an interpreter and gui package manager to stop you jumping to the command line, and a help system to stop you using man. By buying becomming closely associated with R, Microsoft is more likely to be a component in large analytics investments i.e. "let's get Microsoft's R distro, oh and why not their big data plaform too?", and ensures its services and databases are well integrated in this upcoming language.

fixxer 11y ago
"PHP of analytics" I started laughing, then I realized you meant that as a compliment. R has a great community and a ton of unique packages, though many of those great packages are thin wrappers over Fortran/C libraries and are available in a host of other languages (Python, Julia) that, in my opinion, have better features when it comes to testing and group-based development. In my experience, R degenerates into a rat's nest more easily than the other candidates. I've been hired twice for this explicit reason. BTW, I think your comparison with PHP is apt. R has many of the same strengths and flaws.
- taylorwc 11y ago
  Slight typo: compliment is correct here, rather than complement. Excellent discussion.
  - deleted 11y ago
    [deleted]