4 ms·

The write once run anywhere concept is intriguing. R has major scaling issues and a lot of the algorithms and packages in the CRAN just don't scale to any reaso

The write once run anywhere concept is intriguing. R has major scaling issues and a lot of the algorithms and packages in the CRAN just don't scale to any reasonable amounts of data. It really was designed in an era of controlled experiments with minimal data.

mjfl 11y ago
R is a funny language because there are a lot of things in it that are just objectively bad, yet I still love it. Probably because of the community and CRAN. Of course, R is really just some syntax wrapped over C++, and it is very extendable using C and C++ when needed, so I don't think you can really say, in absolute, that R doesn't scale. Someone has written a package to scale whatever problem in R, you just need to find it.
stdbrouw 11y ago
> an era of controlled experiments with minimal data This world hasn't disappeared, y'know. Many of the specialized packages on CRAN deal with things like missing data, removing selection bias from longitudinal studies and distinguishing mediated from direct effects. These concerns are by and large not even applicable to the data the average data scientist deals with, so why even bother making them scalable? On the ML front, on the other hand, R has never really been a forerunner.
- blumkvist 11y ago
  I don't know why this comment is downvoted. It is true. The vast majority of businesses will never deal with machine learning. ML has very limited applications in the "real world", as opposed to regression analysis with some small samples.
  - ska 11y ago
    Your point that simple regressions with small(ish) sample sets are important is valid. This: > "The vast majority of businesses will never deal with machine learning." However, is myopic. It plausibly will not hold for much longer in most areas, and is already not true for a significant number of industries.
    - blumkvist 11y ago
      I find ML and "3d printing" very similar in the expectations and extent business will adopt them.
      - ska 11y ago
        On what basis? They are not similar in any respect that I can think of. 3D printing has had limited impact for prototyping in the many decades it has been around, and in the expected quarters. That's pretty much what every analysis I've seen except a very few boosters have predicted, for what it's worth. On the other hand ML has fundamentally changed: document processing, search engines, remote sensing, retail sales, e-commerce, etc. ; It has also strongly impacted areas of logistics and shipping, manufacturing, (e.g. computer vision & robotics), medicine, law enforcement, finance, defense, also etc. It's also poised to find market advantages in many non-obvious industries as larger scale data becomes available... unclear how far that will extend. Where's the parallel you see?
        IndianAstronaut 11y ago
        Many of the companies I have interviewed and all the ones I have worked for have set up ML and analytics teams in the last 5 years. It is a growing area and people are starting to understand its utility.
        blumkvist 11y ago
        Poorly understood, high hopes, a lot lower practical applications. Predictions are not as applicable as one might think at first glance. I hope you do not confuse machine learning with statistics and data mining. They are very different.
        ska 11y ago
        No, I do not make such a confusion. I hope you do not make the converse one, of re-categorizing techniques "out" of machine learning as soon as we understand how they work. At any rate, your experience in industry seems to be quite different than mine.
nl 11y ago
The work going on in Spark[1] is pretty interesting from this point of view. Using R functions via closures against Spark RDDs in a cluster has the potential to deliver pretty good performance against large scale data. [1] https://amplab-extras.github.io/SparkR-pkg/ https://amplab-extras.github.io/SparkR-pkg/
- IndianAstronaut 11y ago
  I am very excited about the SparkR package. I have it set up on my desktop box and look forward to seeing what models and learning algorithms will work well with it. There really is no runner up to R in terms of the variety of statistical models, now what it needs is to scale up.

mjfl 11y ago

R is a funny language because there are a lot of things in it that are just objectively bad, yet I still love it. Probably because of the community and CRAN. Of course, R is really just some syntax wrapped over C++, and it is very extendable using C and C++ when needed, so I don't think you can really say, in absolute, that R doesn't scale. Someone has written a package to scale whatever problem in R, you just need to find it.

stdbrouw 11y ago

> an era of controlled experiments with minimal data This world hasn't disappeared, y'know. Many of the specialized packages on CRAN deal with things like missing data, removing selection bias from longitudinal studies and distinguishing mediated from direct effects. These concerns are by and large not even applicable to the data the average data scientist deals with, so why even bother making them scalable? On the ML front, on the other hand, R has never really been a forerunner.

blumkvist 11y ago
I don't know why this comment is downvoted. It is true. The vast majority of businesses will never deal with machine learning. ML has very limited applications in the "real world", as opposed to regression analysis with some small samples.
- ska 11y ago
  Your point that simple regressions with small(ish) sample sets are important is valid. This: > "The vast majority of businesses will never deal with machine learning." However, is myopic. It plausibly will not hold for much longer in most areas, and is already not true for a significant number of industries.
  - blumkvist 11y ago
    I find ML and "3d printing" very similar in the expectations and extent business will adopt them.
    - ska 11y ago
      On what basis? They are not similar in any respect that I can think of. 3D printing has had limited impact for prototyping in the many decades it has been around, and in the expected quarters. That's pretty much what every analysis I've seen except a very few boosters have predicted, for what it's worth. On the other hand ML has fundamentally changed: document processing, search engines, remote sensing, retail sales, e-commerce, etc. ; It has also strongly impacted areas of logistics and shipping, manufacturing, (e.g. computer vision & robotics), medicine, law enforcement, finance, defense, also etc. It's also poised to find market advantages in many non-obvious industries as larger scale data becomes available... unclear how far that will extend. Where's the parallel you see?
      - IndianAstronaut 11y ago
        Many of the companies I have interviewed and all the ones I have worked for have set up ML and analytics teams in the last 5 years. It is a growing area and people are starting to understand its utility.
      - blumkvist 11y ago
        Poorly understood, high hopes, a lot lower practical applications. Predictions are not as applicable as one might think at first glance. I hope you do not confuse machine learning with statistics and data mining. They are very different.
        ska 11y ago
        No, I do not make such a confusion. I hope you do not make the converse one, of re-categorizing techniques "out" of machine learning as soon as we understand how they work. At any rate, your experience in industry seems to be quite different than mine.

nl 11y ago

The work going on in Spark[1] is pretty interesting from this point of view. Using R functions via closures against Spark RDDs in a cluster has the potential to deliver pretty good performance against large scale data. [1] https://amplab-extras.github.io/SparkR-pkg/ https://amplab-extras.github.io/SparkR-pkg/

IndianAstronaut 11y ago
I am very excited about the SparkR package. I have it set up on my desktop box and look forward to seeing what models and learning algorithms will work well with it. There really is no runner up to R in terms of the variety of statistical models, now what it needs is to scale up.