4 ms·

So why not just use Python these days?

billwilliams 14y ago
Use both and you'll see. Python can do modeling, can do exploratory work and is great in production. R is brilliant at exploratory work and modeling. Having a core in functional programming helps too.
- tocomment 14y ago
  Actually I just finished my first week working in R. And I could have done the same amount of work in Python in an hour :-( It's like every line I wrote there was some catch or trick I had to know just to get it to work. Pretty frustrating.
  - vsbuffalo 14y ago
    That's just part of learning programming languages. Learn a language well, then decide. I thought Python was better than R, but a homoiconic functional language just makes sense for statistics. You get really neat functions like with, within, local, that let you keep everything tidy. S3/4 classes are hugely inspired by CLOS. It's also a great language for extending C with.
    - houshuang 14y ago
      I've just been learning R for my grad studies (and general data exploration), and was really torn between R and Python, especially with IPython Notebook, pandas etc. I came from a Ruby background so both would be newish to me. What made me start with R (although I'm still following IPython dev, and will probably end up switching eventually) is the massive amount of material available to learn stats etc with R. I've seriously got something like a 100 textbooks on R on my system, intro to stats with R, machine learning with R, questionnaire analysis with R, bayesian stats with R... There are Coursera classes, the amazing r-bloggers.com, etc. For someone who is simultaneously learning statistics and a tool, this is invaluable. For Python, there's still very little. There's Wes' book, which is mostly about pandas and a lot of finance/time series stuff... I haven't seen a single "intro to stats in social sciences with Python"... There must be one book out there that is open source, which could just be rewritten with Python examples? (The most useful thing I've seen is people reworking examples from Machine Learning for Hackers, or one of the Coursera R courses, in Python).
      - phren0logy 14y ago
        There is one basic stats book using Python: http://greenteapress.com/thinkstats/ http://greenteapress.com/thinkstats/
  - rm999 14y ago
    As the article discusses, R has a steep learning curve. Mastering R requires knowing dozens of commands and tricks, but once you learn those the language becomes very expressive and quick to work with. I agree it's frustrating. I've been working with R on and off for more than five years and almost exclusively for 1-2 years, and I still need google every ten minutes to remember some esoteric command that is perfect for the problem at hand. In my experience this is still quicker and less error-prone than python, which often requires you to effectively roll out your own solutions for small data processing needs.
    - trts 14y ago
      I'm about where you are at. I struggled with doing things in R for a year or so, while I was initially using it to do some cool graphics that I couldn't elsewhere, following mostly by example. After spending enough time using it, things began to "click" for me, and now I prefer to do most of my work in R. I still struggle sometimes to write nice looking code in R, but I've known people who are pretty good at managing it. If you have to inherit an R program from someone else though, it's almost guaranteed to be a nightmare to comprehend.
rm999 14y ago
At this point, mostly library support. Python is quickly getting closer to being a viable R replacement, but it's simply not there yet IMO. The two biggest holes in python for me are: * The lack of built-in dataframes and libraries to work on them (like plyr). Pandas seems to be getting pretty good, but it's still not as mature as R's solutions. * Visualization. ggplot for R is great, matplotlib for python not so good IMO. I've heard Bokeh and rplot are attempting to bring ggplot functionality to python. Again, not nearly as mature as R's solution. I'd love to move to Python because R is not a fun language to develop software in. But at this point, R is by far the better tool for working with data (for my needs at least).
- elyase 14y ago
  Any specific example on what pandas is lacking in comparison to R packages?
- hadley 14y ago
  I'd love to hear what makes R not-fun to develop in. I'm always on the look-out for common pain points.
  - rm999 14y ago
    A large pain point for me is remembering the huge number of useful commands in R - I use at least 5-10x as many keywords in R as I did in C++. As I mention in another comment I basically need a reference (usually google and ?) at all times to keep track of these commands and their parameters. In many ways it's great that all these functions exist because I can analyze datasets an order of magnitude more quickly (in development time) than I could in a more traditional language, but developing in R is far more stressful to me. When I'm writing production-level code I constantly have to worry about readability and whether someone else who is reading my code will intuitively understand what I'm doing. With so many commands and so many ways to do things this can be challenging. I find my R code is often lower 'quality' than what I write in something like Python or C. There are many little idiosyncrasies in R's syntax, I feel like I never grokked the language. For example, pretty much anytime I see '~' I have to relearn what is going on. From a mathematical perspective I appreciate that vectors are indexed from 1 instead of 0, but from a programming perspective it can be annoying. BTW, thanks for contributing so many great things to R, I owe a lot of what I do to you.
minimaxir 14y ago
I don't know how Python handles it, but R uses vector operations for almost every function, meaning that for large data sets (10000+), the efficiency of data manipulation in R is always O(1), and may otherwise be O(n) in Python. I have a function in R that given a list of keywords, for each word (~100 keywords), it checks a data frame that stores a list of keywords (~5000 lists of keywords, ~10 keywords/list) to see which keywords matches the keywords in each row of the data frame. Efficiency using straight for loops would be O(mno) and could take awhile. But in R, all those operations are vectors, and it takes less than a second.
- teraflop 14y ago
  "Vector operations" cannot magically turn an O(n) operation into an O(1) operation, no matter what language you're using.
  - minimaxir 14y ago
    Fair enough. You're right, O(1) might not be the most accurate efficiency for what I'm describing.
- pyoung 14y ago
  I imagine your mileage may vary depending on the functions and operations that you are performing. I did a simple test of python vs R and python was the overwhelmingly faster. I believe both languages are primarily written in C, and as such, the speed of your program will depend on the 'efficiency' of the underlying code for the functions you are using.
  - cschmidt 14y ago
    R is horrifically slow on loops. If you can vectorize the operation, then it will be reasonably fast. If you have to use an explicit loop, then python will be faster. My previous company had a saying that "R is fast if your write it in C". If you have many nested loops, you'll probably need to write a C extension.

Use both and you'll see. Python can do modeling, can do exploratory work and is great in production. R is brilliant at exploratory work and modeling. Having a core in functional programming helps too.

tocomment 14y ago
Actually I just finished my first week working in R. And I could have done the same amount of work in Python in an hour :-( It's like every line I wrote there was some catch or trick I had to know just to get it to work. Pretty frustrating.
- vsbuffalo 14y ago
  That's just part of learning programming languages. Learn a language well, then decide. I thought Python was better than R, but a homoiconic functional language just makes sense for statistics. You get really neat functions like with, within, local, that let you keep everything tidy. S3/4 classes are hugely inspired by CLOS. It's also a great language for extending C with.
  - houshuang 14y ago
    I've just been learning R for my grad studies (and general data exploration), and was really torn between R and Python, especially with IPython Notebook, pandas etc. I came from a Ruby background so both would be newish to me. What made me start with R (although I'm still following IPython dev, and will probably end up switching eventually) is the massive amount of material available to learn stats etc with R. I've seriously got something like a 100 textbooks on R on my system, intro to stats with R, machine learning with R, questionnaire analysis with R, bayesian stats with R... There are Coursera classes, the amazing r-bloggers.com, etc. For someone who is simultaneously learning statistics and a tool, this is invaluable. For Python, there's still very little. There's Wes' book, which is mostly about pandas and a lot of finance/time series stuff... I haven't seen a single "intro to stats in social sciences with Python"... There must be one book out there that is open source, which could just be rewritten with Python examples? (The most useful thing I've seen is people reworking examples from Machine Learning for Hackers, or one of the Coursera R courses, in Python).
    - phren0logy 14y ago
      There is one basic stats book using Python: http://greenteapress.com/thinkstats/ http://greenteapress.com/thinkstats/
- rm999 14y ago
  As the article discusses, R has a steep learning curve. Mastering R requires knowing dozens of commands and tricks, but once you learn those the language becomes very expressive and quick to work with. I agree it's frustrating. I've been working with R on and off for more than five years and almost exclusively for 1-2 years, and I still need google every ten minutes to remember some esoteric command that is perfect for the problem at hand. In my experience this is still quicker and less error-prone than python, which often requires you to effectively roll out your own solutions for small data processing needs.
  - trts 14y ago
    I'm about where you are at. I struggled with doing things in R for a year or so, while I was initially using it to do some cool graphics that I couldn't elsewhere, following mostly by example. After spending enough time using it, things began to "click" for me, and now I prefer to do most of my work in R. I still struggle sometimes to write nice looking code in R, but I've known people who are pretty good at managing it. If you have to inherit an R program from someone else though, it's almost guaranteed to be a nightmare to comprehend.

rm999 14y ago

At this point, mostly library support. Python is quickly getting closer to being a viable R replacement, but it's simply not there yet IMO. The two biggest holes in python for me are: * The lack of built-in dataframes and libraries to work on them (like plyr). Pandas seems to be getting pretty good, but it's still not as mature as R's solutions. * Visualization. ggplot for R is great, matplotlib for python not so good IMO. I've heard Bokeh and rplot are attempting to bring ggplot functionality to python. Again, not nearly as mature as R's solution. I'd love to move to Python because R is not a fun language to develop software in. But at this point, R is by far the better tool for working with data (for my needs at least).

elyase 14y ago
Any specific example on what pandas is lacking in comparison to R packages?
hadley 14y ago
I'd love to hear what makes R not-fun to develop in. I'm always on the look-out for common pain points.
- rm999 14y ago
  A large pain point for me is remembering the huge number of useful commands in R - I use at least 5-10x as many keywords in R as I did in C++. As I mention in another comment I basically need a reference (usually google and ?) at all times to keep track of these commands and their parameters. In many ways it's great that all these functions exist because I can analyze datasets an order of magnitude more quickly (in development time) than I could in a more traditional language, but developing in R is far more stressful to me. When I'm writing production-level code I constantly have to worry about readability and whether someone else who is reading my code will intuitively understand what I'm doing. With so many commands and so many ways to do things this can be challenging. I find my R code is often lower 'quality' than what I write in something like Python or C. There are many little idiosyncrasies in R's syntax, I feel like I never grokked the language. For example, pretty much anytime I see '~' I have to relearn what is going on. From a mathematical perspective I appreciate that vectors are indexed from 1 instead of 0, but from a programming perspective it can be annoying. BTW, thanks for contributing so many great things to R, I owe a lot of what I do to you.

minimaxir 14y ago

I don't know how Python handles it, but R uses vector operations for almost every function, meaning that for large data sets (10000+), the efficiency of data manipulation in R is always O(1), and may otherwise be O(n) in Python. I have a function in R that given a list of keywords, for each word (~100 keywords), it checks a data frame that stores a list of keywords (~5000 lists of keywords, ~10 keywords/list) to see which keywords matches the keywords in each row of the data frame. Efficiency using straight for loops would be O(mno) and could take awhile. But in R, all those operations are vectors, and it takes less than a second.

teraflop 14y ago
"Vector operations" cannot magically turn an O(n) operation into an O(1) operation, no matter what language you're using.
- minimaxir 14y ago
  Fair enough. You're right, O(1) might not be the most accurate efficiency for what I'm describing.
pyoung 14y ago
I imagine your mileage may vary depending on the functions and operations that you are performing. I did a simple test of python vs R and python was the overwhelmingly faster. I believe both languages are primarily written in C, and as such, the speed of your program will depend on the 'efficiency' of the underlying code for the functions you are using.
- cschmidt 14y ago
  R is horrifically slow on loops. If you can vectorize the operation, then it will be reasonably fast. If you have to use an explicit loop, then python will be faster. My previous company had a saying that "R is fast if your write it in C". If you have many nested loops, you'll probably need to write a C extension.