6 ms·

The article says this particular instance of the debate started in 2011. Things have shifted a little since then, and I think Python has won more mindshare wit

by jzwinck 13y ago

The article says this particular instance of the debate started in 2011. Things have shifted a little since then, and I think Python has won more mindshare with Pandas, SciPy, NumPy, and all the rest. I've used both Python and R, and think the next debate will be between those two, as people find that R is not a very good programming language and lacks decent libraries for things like web scraping. Python can be a single tool that integrates with every part of your workflow. R right now still wins in the number of algorithms implemented in it (there are statistical methods not available in R but not Python), and R has more terse syntax which some people like for interactive use. But for really Big Data, terse syntax and an endless variety of esoteric algorithms are not as important as, say, robust error handling and debugging (a weak area in R, but a strong one in Python).

krick 13y ago
> R right now still wins in the number of algorithms implemented in it Can you please list some important tools implemented in R but not in Python? I'd like to know how bit the gap is.
- mjfl 13y ago
  Is there an equivalent Python package to R's {stats}? http://stat.ethz.ch/R-manual/R-patched/library/stats/html/00Index.html http://stat.ethz.ch/R-manual/R-patched/library/stats/html/00...
  - jofer 13y ago
    Have a look at statsmodels: http://statsmodels.sourceforge.net/ http://statsmodels.sourceforge.net/ and for more basic things, scipy.stats: http://docs.scipy.org/doc/scipy/reference/stats.html http://docs.scipy.org/doc/scipy/reference/stats.html It's not a one-to-one match, but the majority of the functionality is there.
- Fomite 13y ago
  I think the biggest problem is for most of my statistical work, for Python I'd have to check, for R I can safely simply assume an implementation (or several) exists.
- gwern 13y ago
  So, I'm not a statistician but I do wind up using a lot of (R) statistics tools in my various writings: http://www.gwern.net/tags/statistics http://www.gwern.net/tags/statistics R is a terrible terrible language from my perspective as a Haskeller, but the libraries are just too fantastic for me to not use it. A crude bit of shell (`fgrep --no-filename 'library(' .page|sed -e 's/library//'|tr -d ')'|tr -d '('|sort -u`) on my writings and a bit of cleanup gets me the following list of R libraries I use: MASS MKmisc Rcapture XML arm biglm boot changepoint ggplot2 insol languageR lme4 lubridate metafor moonsun prodlim pwr randomForest randomForestSRC reshape2 rjson rms stringr survival verification Some of them aren't important and I'm sure Python has equivalents to things like 'rjson', but others...? Let's look at a few. - metafor: meta-analysis library (fixed & random-effects, forest plots, bias checks, etc). I use this in 3 meta-analyses I've done, so it's kind of important. A google of 'python meta-analysis' turns up... nothing? Well, it turns up some Bayesian stuff, but nothing jumps out as a finished polished library that one could easily reimplement my work in. ('METASOFT' sounds promising, but actually turns out to be for genomics and in Java.) - randomForest, randomForestSRC: random forests are apparently implemented by Scikit. Great. What about random survival* forests, which I use in my Google shutdowns analysis to see if the standard survival model can be improved upon? 5th hit for 'python random survival forests' is... an R library. - Rcapture: capture-recapture analysis, which I employ in my hafu analysis of anime/manga to try to estimate how many I may be missing. 'python capture-recapture analysis': the best-looking hit (http://www.mikemeredith.net/ http://www.mikemeredith.net/) is a false-positive. The relevant code is - you guessed it - in R. - biglm: a package for incrementally updating a linear model, the idea being that you can load into memory as much data as fits, update the linear model, and load the next batch. I need this sucker for running even simple analyses on the Mnemosyne spaced repetition data set, where the database (~18GB) won't fit in my laptop's RAM. Some googling suggests that Scikit may support this. Or may not. - lme4: the most famous library for doing multilevel models (indispensable generalizations of linear models, great for dealing with multiple measurements from the same subject; I use them on psychology or sleep data, usually). All the Python alternatives appear to either be exercises or Bayesian. - survival: survival analysis library, used for Google shutdowns. https://stats.stackexchange.com/questions/1736/survival-analysis-tools-in-python https://stats.stackexchange.com/questions/1736/survival-anal... says you can sorta do them in Python, but you're cautioned against it and it looks like the Python alternatives don't have the Cox proportional hazards needed to use covariates. I could go on to look at the others (any ordinal logistic regression in Python? how're the bootstrap libraries? what if I need to do power calculations for standard tests?), but I hope this gives you an idea of why I'm not doing everything in Python.
robertk 13y ago
What are you talking about? R has some of the most robust debugging tools I have encountered. You should look into the functions: browser, traceback, recover, trace, dump.frames, debug, debugonce, `environment<-`. What other language allows you to insert breakpoints (or arbitrary code) in the middle of a function's body without even having to re-paste the source code into the console? If you're unsatisfied with R's debugging tools, you are probably not familiar with them: http://adv-r.had.co.nz/Exceptions-Debugging.html http://adv-r.had.co.nz/Exceptions-Debugging.html
robertk 13y ago
Disagree. I've had an R server running in production for real-time analytics under heavy load in the last six months, and I have had exactly zero problems. Literally no down time. Additionally, weak scraping support is a myth: http://www.theswarmlab.com/r-vs-python-round-2-22/ http://www.theswarmlab.com/r-vs-python-round-2-22/
- gwu78 13y ago
  I use a TCP client written by another UIC math professor and sed for most of my scraping needs. Can you or anyone give me an example, _that is likely to be seen on any website_, that you believe cannot be solved with sed but can be solved with XPath? The example at the URL you given above appears to have a target CSV file as its only requirement. I routinely use HTTP pipelining and sed/tr to produce CSV files from websites. As for R, I'm more curious about its database and analytics capabilities. How does it stack up to kdb+?
Fomite 13y ago
> The article says this particular instance of the debate started in 2011. Things have shifted a little since then, and I think Python has won more mindshare with Pandas, SciPy, NumPy, and all the rest. In my field this isn't at all true - R is eroding SAS's market share, but Python is a non-entity.

by jzwinck 13y ago

krick 13y ago

> R right now still wins in the number of algorithms implemented in it Can you please list some important tools implemented in R but not in Python? I'd like to know how bit the gap is.

mjfl 13y ago
Is there an equivalent Python package to R's {stats}? http://stat.ethz.ch/R-manual/R-patched/library/stats/html/00Index.html http://stat.ethz.ch/R-manual/R-patched/library/stats/html/00...
- jofer 13y ago
  Have a look at statsmodels: http://statsmodels.sourceforge.net/ http://statsmodels.sourceforge.net/ and for more basic things, scipy.stats: http://docs.scipy.org/doc/scipy/reference/stats.html http://docs.scipy.org/doc/scipy/reference/stats.html It's not a one-to-one match, but the majority of the functionality is there.
Fomite 13y ago
I think the biggest problem is for most of my statistical work, for Python I'd have to check, for R I can safely simply assume an implementation (or several) exists.
gwern 13y ago
So, I'm not a statistician but I do wind up using a lot of (R) statistics tools in my various writings: http://www.gwern.net/tags/statistics http://www.gwern.net/tags/statistics R is a terrible terrible language from my perspective as a Haskeller, but the libraries are just too fantastic for me to not use it. A crude bit of shell (`fgrep --no-filename 'library(' .page|sed -e 's/library//'|tr -d ')'|tr -d '('|sort -u`) on my writings and a bit of cleanup gets me the following list of R libraries I use: MASS MKmisc Rcapture XML arm biglm boot changepoint ggplot2 insol languageR lme4 lubridate metafor moonsun prodlim pwr randomForest randomForestSRC reshape2 rjson rms stringr survival verification Some of them aren't important and I'm sure Python has equivalents to things like 'rjson', but others...? Let's look at a few. - metafor: meta-analysis library (fixed & random-effects, forest plots, bias checks, etc). I use this in 3 meta-analyses I've done, so it's kind of important. A google of 'python meta-analysis' turns up... nothing? Well, it turns up some Bayesian stuff, but nothing jumps out as a finished polished library that one could easily reimplement my work in. ('METASOFT' sounds promising, but actually turns out to be for genomics and in Java.) - randomForest, randomForestSRC: random forests are apparently implemented by Scikit. Great. What about random survival* forests, which I use in my Google shutdowns analysis to see if the standard survival model can be improved upon? 5th hit for 'python random survival forests' is... an R library. - Rcapture: capture-recapture analysis, which I employ in my hafu analysis of anime/manga to try to estimate how many I may be missing. 'python capture-recapture analysis': the best-looking hit (http://www.mikemeredith.net/ http://www.mikemeredith.net/) is a false-positive. The relevant code is - you guessed it - in R. - biglm: a package for incrementally updating a linear model, the idea being that you can load into memory as much data as fits, update the linear model, and load the next batch. I need this sucker for running even simple analyses on the Mnemosyne spaced repetition data set, where the database (~18GB) won't fit in my laptop's RAM. Some googling suggests that Scikit may support this. Or may not. - lme4: the most famous library for doing multilevel models (indispensable generalizations of linear models, great for dealing with multiple measurements from the same subject; I use them on psychology or sleep data, usually). All the Python alternatives appear to either be exercises or Bayesian. - survival: survival analysis library, used for Google shutdowns. https://stats.stackexchange.com/questions/1736/survival-analysis-tools-in-python https://stats.stackexchange.com/questions/1736/survival-anal... says you can sorta do them in Python, but you're cautioned against it and it looks like the Python alternatives don't have the Cox proportional hazards needed to use covariates. I could go on to look at the others (any ordinal logistic regression in Python? how're the bootstrap libraries? what if I need to do power calculations for standard tests?), but I hope this gives you an idea of why I'm not doing everything in Python.

robertk 13y ago

What are you talking about? R has some of the most robust debugging tools I have encountered. You should look into the functions: browser, traceback, recover, trace, dump.frames, debug, debugonce, `environment<-`. What other language allows you to insert breakpoints (or arbitrary code) in the middle of a function's body without even having to re-paste the source code into the console? If you're unsatisfied with R's debugging tools, you are probably not familiar with them: http://adv-r.had.co.nz/Exceptions-Debugging.html http://adv-r.had.co.nz/Exceptions-Debugging.html

robertk 13y ago

Disagree. I've had an R server running in production for real-time analytics under heavy load in the last six months, and I have had exactly zero problems. Literally no down time. Additionally, weak scraping support is a myth: http://www.theswarmlab.com/r-vs-python-round-2-22/ http://www.theswarmlab.com/r-vs-python-round-2-22/

gwu78 13y ago
I use a TCP client written by another UIC math professor and sed for most of my scraping needs. Can you or anyone give me an example, _that is likely to be seen on any website_, that you believe cannot be solved with sed but can be solved with XPath? The example at the URL you given above appears to have a target CSV file as its only requirement. I routinely use HTTP pipelining and sed/tr to produce CSV files from websites. As for R, I'm more curious about its database and analytics capabilities. How does it stack up to kdb+?

Fomite 13y ago

> The article says this particular instance of the debate started in 2011. Things have shifted a little since then, and I think Python has won more mindshare with Pandas, SciPy, NumPy, and all the rest. In my field this isn't at all true - R is eroding SAS's market share, but Python is a non-entity.