7 ms·

R in production systems

39 points by papps 16y ago

thangalin 16y ago
The Climate Reports (http://www.whitemagicsoftware.com/software/climate http://www.whitemagicsoftware.com/software/climate) has been running for 10 months, without a single failure. The Climate Reports uses R to calculate the trend line. During development, I encountered a bug with one of the packages. I told the author and he not only fixed the bug within 24 hours but also offered to create a custom build for my CPU architecture. I have only good things to say about R and its community. And I think R is sufficiently stable for production.
chillaxn 16y ago
I have been reconsidering my workflow for a more data-centric approach. When I first looked around the search results for data-centric workflow, I was shocked that it was limited mostly to theoretical discussions and commentaries. Why don't we have communities online focussed around using tools and platforms that allow informed discussion and decision making to solve entrepreneurial problems??
secret 16y ago
The advice to go 64-bit and throw RAM at the problem brings back memories of my computational math class.
pjscott 16y ago
I believe this is one of the problems that Incanter was made for: really easy embedding in a larger software system. Of course, it's nowhere near as featureful or widely-used as R. http://incanter.org/ http://incanter.org/
wesm 16y ago
R is a stable enough language to be used successfully in production systems. But I would definitely not recommend building anything "big" in R-- I've tried and the entire process was not pleasant. R is a research environment for statisticians or statistically-minded practitioners in other fields-- and there has not been terribly much effort (from what I can tell) to change its status. I'd be happy to hear from others who've had similar (or different, perhaps positive) experiences. It's actually my experiences trying to build production statistical processes in R that have motivated me to work on building out the statistics-related libraries for scientific Python. Hopefully within a couple of years we'll have something that's competitive with R in terms of ease-of-use and comprehensive statistical functionality out-of-the-box.
- chillaxn 16y ago
  Do you think that Python libraries or a comprehensive development push will overcome barriers to entry for statistical process platforms? What if the greater adoption of these techniques is more limited by communities of users than the developers?
  - wesm 16y ago
    Depends on what you're referring to by "statistical process platforms". Python has been quite popular in the machine learning communities for several years, with some pure Python libraries (e.g. nltk, scikits.learn), and many wrappers to non-Python libraries (Orange, Shogun, and I'm sure some others). One issue at the moment is that there's an R package for just about anything you would want to do. But the issue with R is its software development tools (which are not very good) and its ability to integrate with other systems. As far as mainstream frequentist statistics in Python, there's still a ways to go. scikits.statsmodels has made a lot of headway the last couple years in implementing standard linear (regression) modeling classes and econometrics tools, with some time series analysis and other things thrown in (I'm actually actively working on adding time series analysis functionality to that library). My general view is that given how friendly a language Python is for scientific programmers and the vibrancy of the community (in terms of new tools being built, etc.), Python could become a permanent fixture for statistical computing. Add on top of that things like integrated distributed computing inside IPython and projects like PyCUDA (which could make high performance statistical computing much more accessible), it's looking very good. But there's a slight chicken-and-egg problem at the moment. I should also point out that R is sort of a frankenstein of half-baked object oriented ideas and remnants of the Splus era. So if you want to design extensible classes for statistical models, it can be quite challenging.
    - chillaxn 16y ago
      Not being a scientific programmer, I'm not exposed to the vibrancy of the community that you refer to. Would you recommend any websites where I could follow some discussions on the topic to get a better understand of the available community resources? I will look into Python and its libraries for the research I perform.
mattwalker62 16y ago
good article... I have just launched a site (http://www.promepi.com http://www.promepi.com) which uses R to recommend news articles to users. My co-founder used a lot when getting his masters degree, but I was new to it. It has been great to quickly develop our algorithm. Anyone have experience with Revolution Analytics? My cofounder looked at these guys as we try to scale the site... he definitely agrees with one of the comments that "big" datasets are good in R and hopefully our site gets to a big dataset, so we are looking at scaling options.
ShabbyDoo 16y ago
ESRI (The geospatial software company) and SAS (the commercial alternative to R) both got caught up reacting to their existing user bases instead of proactively watching the analytics world. Both companies have (or at least until recently had as my knowledge is circa-2008) very desktop-centric product lines. Their customers used their products as if they were fancy versions of Excel -- a perfectly valid use case, but not one suited to production analytics. Both companies first forays into production systems involved clunky wrappers around desktop components. I'm sure they will eventually put out first-rate headless production components, but there's probably a nice window available in which start-ups may innovate.

39 points by papps 16y ago

The Climate Reports (http://www.whitemagicsoftware.com/software/climate http://www.whitemagicsoftware.com/software/climate) has been running for 10 months, without a single failure. The Climate Reports uses R to calculate the trend line. During development, I encountered a bug with one of the packages. I told the author and he not only fixed the bug within 24 hours but also offered to create a custom build for my CPU architecture. I have only good things to say about R and its community. And I think R is sufficiently stable for production.

chillaxn 16y ago

I have been reconsidering my workflow for a more data-centric approach. When I first looked around the search results for data-centric workflow, I was shocked that it was limited mostly to theoretical discussions and commentaries. Why don't we have communities online focussed around using tools and platforms that allow informed discussion and decision making to solve entrepreneurial problems??

secret 16y ago

The advice to go 64-bit and throw RAM at the problem brings back memories of my computational math class.

pjscott 16y ago

I believe this is one of the problems that Incanter was made for: really easy embedding in a larger software system. Of course, it's nowhere near as featureful or widely-used as R. http://incanter.org/ http://incanter.org/

wesm 16y ago

R is a stable enough language to be used successfully in production systems. But I would definitely not recommend building anything "big" in R-- I've tried and the entire process was not pleasant. R is a research environment for statisticians or statistically-minded practitioners in other fields-- and there has not been terribly much effort (from what I can tell) to change its status. I'd be happy to hear from others who've had similar (or different, perhaps positive) experiences. It's actually my experiences trying to build production statistical processes in R that have motivated me to work on building out the statistics-related libraries for scientific Python. Hopefully within a couple of years we'll have something that's competitive with R in terms of ease-of-use and comprehensive statistical functionality out-of-the-box.

chillaxn 16y ago
Do you think that Python libraries or a comprehensive development push will overcome barriers to entry for statistical process platforms? What if the greater adoption of these techniques is more limited by communities of users than the developers?
- wesm 16y ago
  Depends on what you're referring to by "statistical process platforms". Python has been quite popular in the machine learning communities for several years, with some pure Python libraries (e.g. nltk, scikits.learn), and many wrappers to non-Python libraries (Orange, Shogun, and I'm sure some others). One issue at the moment is that there's an R package for just about anything you would want to do. But the issue with R is its software development tools (which are not very good) and its ability to integrate with other systems. As far as mainstream frequentist statistics in Python, there's still a ways to go. scikits.statsmodels has made a lot of headway the last couple years in implementing standard linear (regression) modeling classes and econometrics tools, with some time series analysis and other things thrown in (I'm actually actively working on adding time series analysis functionality to that library). My general view is that given how friendly a language Python is for scientific programmers and the vibrancy of the community (in terms of new tools being built, etc.), Python could become a permanent fixture for statistical computing. Add on top of that things like integrated distributed computing inside IPython and projects like PyCUDA (which could make high performance statistical computing much more accessible), it's looking very good. But there's a slight chicken-and-egg problem at the moment. I should also point out that R is sort of a frankenstein of half-baked object oriented ideas and remnants of the Splus era. So if you want to design extensible classes for statistical models, it can be quite challenging.
  - chillaxn 16y ago
    Not being a scientific programmer, I'm not exposed to the vibrancy of the community that you refer to. Would you recommend any websites where I could follow some discussions on the topic to get a better understand of the available community resources? I will look into Python and its libraries for the research I perform.

mattwalker62 16y ago

good article... I have just launched a site (http://www.promepi.com http://www.promepi.com) which uses R to recommend news articles to users. My co-founder used a lot when getting his masters degree, but I was new to it. It has been great to quickly develop our algorithm. Anyone have experience with Revolution Analytics? My cofounder looked at these guys as we try to scale the site... he definitely agrees with one of the comments that "big" datasets are good in R and hopefully our site gets to a big dataset, so we are looking at scaling options.

ShabbyDoo 16y ago

ESRI (The geospatial software company) and SAS (the commercial alternative to R) both got caught up reacting to their existing user bases instead of proactively watching the analytics world. Both companies have (or at least until recently had as my knowledge is circa-2008) very desktop-centric product lines. Their customers used their products as if they were fancy versions of Excel -- a perfectly valid use case, but not one suited to production analytics. Both companies first forays into production systems involved clunky wrappers around desktop components. I'm sure they will eventually put out first-rate headless production components, but there's probably a nice window available in which start-ups may innovate.