6 ms·

Goodbye R scripts, hello R packages

51 points by RA_Fisher 13y ago

vikp 13y ago
I love R, and it was actually the first language I really learned to program with (for obvious reasons, I wouldn't ever recommend this). I can identify a lot with the "one-off scripts" problem of R. When I look back at some of my R code, I find that it is just a giant mess of commands mixed together semi-randomly. As I learned to program properly, I solved the reuse and "good code" problem more by moving towards using Python than by making R packages. I occasionally use R for data exploration and visualization, but Python has support for almost all of the machine learning and statistical functions that I need. I am very interested to hear how other people have solved this problem. Do you only use R, or R in combination with Python/Julia/Java? If you only use R, are you using it purely academically?
- RA_Fisher 13y ago
  I'm just now getting into Python. Primarily because I'm abandoning Fisherian methods for Bayesian methods and all of the books seem to have examples in Python. I use R in a business setting. I'm a data scientist at Treehouse (teamtreehouse.com)
  - jasonpbecker 13y ago
    There are great tools that interface R with BUGS that may be worth checking out. I would highly recommend getting this for the shelf: http://www.amazon.com/Analysis-Regression-Multilevel-Hierarchical-Models/dp/052168689X http://www.amazon.com/Analysis-Regression-Multilevel-Hierarc... It's one of the most readable books on data analysis I've come across and does a great job presenting both frequentist and Bayesian techniques with tons of R sample code. There are a lot of advantages and nice things in Python, but I do think folks tend to toss out R a bit too casually. Each tool has areas they excel in. I don't even do particularly complex analysis, but have run into areas where Python is woefully lacking in fairly common (social science) models.
    - RA_Fisher 13y ago
      Thanks! I'm planning on digging into Cam Davidson-Pilon's Bayesian Methods for Hackers soon:https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for- https://github.com/CamDavidsonPilon/Probabilistic-Programmin... Thanks for the link to the book. I could see it being really helpful to me to have frequentist examples alongside Bayesian examples. I'm hoping to learn Python so I have both tools. R is so domain-specific that I doubt I'll ever completely stop using it in my career. Who knows, though!?
- chubot 13y ago
  IMO the Unix philosophy of reuse is vital for data analysis. I recommend that people in this space learn to use the interactive shell and shell scripts well. I saw a comment that said "Shell is a REPL for C", which I think is quite pithy. People think too much about reusing R packages or Python packages. In my experience you can get things done a lot faster if you factor things into programs and not libraries. This lets you use multiple languages, and all real world analysis problems need multiple languages. (If you're only using one language, then you're likely only working on part of the problem). Right my toolset is Python, C++, and R, coordinated with shell scripts. I still like R for quick plotting, and of course data frames are essential, but I'm playing with Pandas now, which seems impressive. I don't think Python will ever catch up to R in terms of statistical functions, but in terms of plotting/munging/data frames it might. In the future there will be more languages, not less (Julia will add to the number of languages, not replace any). So being able to decompose a problem into separate, reasonably generic, programs is an important skill IMO. Most data analysis pipelines are a huge mess, but they don't have to be.
  - vikp 13y ago
    That's an interesting way to go about it. Any reason for using shell scripts to coordinate the flow instead of things like Cython and RPy? (I don't shell script a lot, so this may be a silly question) These days, I mostly seem to be able to get away with using single languages for applied machine learning, like a Python webserver that runs background machine learning tasks, or an Android app in Java that connects to a webserver written in Python. But back when I was doing less applied side stuff, I was similar to you in that things were spread across R and Python.
    - LukeShu 13y ago
      The Unix Shell is basically an IPC language. It's sole purpose is to be able to combine programs, and allow them to communicate. Instead of trying to make a package in one language work with another, it is a lot simpler to use Shell to connect together programs in any language.
    - chubot 13y ago
- vdaniuk 13y ago
  Should one learn R if one already has some experience in data analysis with Python?
  - collyw 13y ago
    I would like to know the same. I work with a lot of bioinformaticians. The ones from a biology background (i.e. code quality is not usually important) seem to like R. The ones with a computer science background seem to go for Python, or Perl if they are old school. So far that has put me off R.
- hadley 13y ago
  I think R is actually a rather great first language, as the focus is usually on doing cool stuff, rather than developing a rigorous framework for programming. It's more important to put motivation before rigour, since if you focus only on rigour most people will lose interest before getting to the fun stuff.
  - vikp 13y ago
    Great point. I still wish I coded in R more, just because the MTTC (mean time to cool) is so much lower. But, I feel like you can do the same with sklearn/ipython notebook/pandas if you ignore the parts of the "rigorous framework", and just focus on the syntax for interesting things.
- oddthink 13y ago
  I'm slowly moving my elaborate data-cleaning system from R to python. I never got the hang of R packages and ended up rolling my own "import" scheme, but now it all seems like I've passed some critical complexity point. I'm still doing all of my model estimation in R, and data.table + ggplot2 is still my go-to solution for interactive exploration and plotting, but I'm putting more and more of the process-csv, write-hdf5, remove out-of-bounds values, backfill, stratified sampling/bootstrap, etc., logic into python. I may end up moving more of the estimation itself into python, now that statsmodels and sklearn are more mature, but R does that pretty well. I've not tried Julia, but I have experimented with Java for some of the data-processing, but it's never seemed worth it, over python. Basic CSV processing is faster in python than in Java (although the python version uses much more CPU), so I haven't felt the need. (I'm probably limited more by disk bandwidth than CPU, but I've not done the tests required to prove that.) Julia seems nice, but it seems intent on copying all the questionable design choices of matlab, rather than using ideas from the numpy/kdb+/apl worlds.
EpiMath 13y ago
Interesting, thanks. ( Like your username too, despite him being slightly denigrated in the linked article. ) I personally think that it is good to have a basic grounding in some kind of fundamental "real" programming language before you get into s-plus ( R ) or any other more domain-specific language. The main reason being that you will better understand the compromises, limitations and shortcomings. I like R but as a programmer I've seen some awful code written in it!
- RA_Fisher 13y ago
  Yes, if you read further on the blog, you'll see I'm more than slightly denigrating. I read a book recently called The Cult of Statistical Significance and it's really turned me against Fisher. Not only do I see his methods as sub-standard to Bayesian analysis, he was really mean and disparaging himself to those folks. It's interesting, my username is what it is because he used to be my hero! Just 6 months ago, I really considered him to be the father of modern science. Now I basically see his work as mostly enabling mediocre scientists to rise through the ranks. This is what I mean by that: http://www.economist.com/blogs/graphicdetail/2013/10/daily-chart-2 http://www.economist.com/blogs/graphicdetail/2013/10/daily-c...
  - EpiMath 13y ago
    I've spent my career thinking about these same issues. And certainly Fisher was, well, abrasive to put it mildly. I've read much of his early work and I think the intended context has mostly been lost. Many modern statisticians ( of the "frequentist" persuasion ) use a strange and awkward combination of fisherian and neyman-pearson. We talk about "p-values" but then interpret them as hypothesis tests with long-term error probabilities ( Fisher disavowed this interpretation of p-values, and insisted they did not have a long run probabilistic interpretation but were a measure only of evidence against the null in the particular experiment. ) I think Fisher gets a bad rap for a lot of later bastardization of his work. ( I'm sure it did not help that he was not a likable person, or his valid but misguided and testimony about tobacco. ) Still, I'm sympathetic to your position and can understand how you'd come to that way of thinking. I'm not convinced even a diehard Bayesian would completely disagree with Fisher's more restricted and stringent interpretation of p-values. Or at least they would see it as a big step up from the more common usage you are referencing.
elohesra 13y ago
Purely out of interest, what's so wrong with Windows as a development environment? I've seen this claim touted before on HN, but I've never seen an explanation (not doubting that there is one; I just haven't seen one). I develop on Windows every day, and I'm always happy to learn a new way to improve my workflow, so does Ubuntu (or Mac) actually make it easier to develop software, and if so then how? I'd also be interested to hear what kinds of software you've found it hard to develop on Windows.
- Xophmeister 13y ago
  In my (anecdotal) experience: A lot of tools (everything from editors to compilers, etc.) that are open source are from the Unix world. They're designed to work in a POSIX way and the ports to Windows (if they exist, or you're forced to use Cygwin) are a bit cludgy: You might just as well have used some kind of Unix in the first place. Windows dev tools are available, of course, but you often have to pay quite a lot for them. Good OSS exists for Windows, but it's a subset of what's readily available under nix, which doesn't encourage the hacker/hobbyist spirit.
  - elohesra 13y ago
    Thanks for the reply. That does make sense. As a C# developer, I haven't really had call to use OSS IDEs, so I can't really comment, but I do remember having an ugly time trying to use an OSS interactive disassembler on Windows.
- CraigJPerry 13y ago
  >> what's so wrong with Windows as a development environment? Command line. When you're doing a task on a computer, some paradigms are better suited than others depending on the task. E.g. imagine delicately touching up pixels in a photo, guided by your artistic eye yet instructing the computer by laboriously typing many commands into a command line. There's no reason this couldn't work, it's just way more natural to use a pen & digtiser, or a mouse. When we're developing we're editing text and invoking commands. Editing text is covered by any major platform just fine, a lot of popular editors & IDEs are even cross platform. Invoking commands are sometimes covered by the IDE, sometimes the commands covered are pretty comprehensive (i'm thinking of Tom Christiansen's quip "Emacs is a nice operating system, but I prefer UNIX"). There's a limit to what tasks your IDE can cover, even though some do try really hard to cover all your needs (Eclipse has a web browser in it!) and when you hit that limit as we so often do in development, you need a command line. At that point, Windows kind of whistles while sheepishly looking off to one side. There's a related point about development dependencies, some libraries are hard and / or time consuming to build. A great package manager is a brilliant productivity booster. Mostly though, it's the point about the command line. (just to be clear, i have given powershell a good shot, i estimate i've authored almost 1k lines of it).

51 points by RA_Fisher 13y ago

vikp 13y ago

I love R, and it was actually the first language I really learned to program with (for obvious reasons, I wouldn't ever recommend this). I can identify a lot with the "one-off scripts" problem of R. When I look back at some of my R code, I find that it is just a giant mess of commands mixed together semi-randomly. As I learned to program properly, I solved the reuse and "good code" problem more by moving towards using Python than by making R packages. I occasionally use R for data exploration and visualization, but Python has support for almost all of the machine learning and statistical functions that I need. I am very interested to hear how other people have solved this problem. Do you only use R, or R in combination with Python/Julia/Java? If you only use R, are you using it purely academically?

RA_Fisher 13y ago
I'm just now getting into Python. Primarily because I'm abandoning Fisherian methods for Bayesian methods and all of the books seem to have examples in Python. I use R in a business setting. I'm a data scientist at Treehouse (teamtreehouse.com)
- jasonpbecker 13y ago
  There are great tools that interface R with BUGS that may be worth checking out. I would highly recommend getting this for the shelf: http://www.amazon.com/Analysis-Regression-Multilevel-Hierarchical-Models/dp/052168689X http://www.amazon.com/Analysis-Regression-Multilevel-Hierarc... It's one of the most readable books on data analysis I've come across and does a great job presenting both frequentist and Bayesian techniques with tons of R sample code. There are a lot of advantages and nice things in Python, but I do think folks tend to toss out R a bit too casually. Each tool has areas they excel in. I don't even do particularly complex analysis, but have run into areas where Python is woefully lacking in fairly common (social science) models.
  - RA_Fisher 13y ago
    Thanks! I'm planning on digging into Cam Davidson-Pilon's Bayesian Methods for Hackers soon:https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for- https://github.com/CamDavidsonPilon/Probabilistic-Programmin... Thanks for the link to the book. I could see it being really helpful to me to have frequentist examples alongside Bayesian examples. I'm hoping to learn Python so I have both tools. R is so domain-specific that I doubt I'll ever completely stop using it in my career. Who knows, though!?
chubot 13y ago
IMO the Unix philosophy of reuse is vital for data analysis. I recommend that people in this space learn to use the interactive shell and shell scripts well. I saw a comment that said "Shell is a REPL for C", which I think is quite pithy. People think too much about reusing R packages or Python packages. In my experience you can get things done a lot faster if you factor things into programs and not libraries. This lets you use multiple languages, and all real world analysis problems need multiple languages. (If you're only using one language, then you're likely only working on part of the problem). Right my toolset is Python, C++, and R, coordinated with shell scripts. I still like R for quick plotting, and of course data frames are essential, but I'm playing with Pandas now, which seems impressive. I don't think Python will ever catch up to R in terms of statistical functions, but in terms of plotting/munging/data frames it might. In the future there will be more languages, not less (Julia will add to the number of languages, not replace any). So being able to decompose a problem into separate, reasonably generic, programs is an important skill IMO. Most data analysis pipelines are a huge mess, but they don't have to be.
- vikp 13y ago
  That's an interesting way to go about it. Any reason for using shell scripts to coordinate the flow instead of things like Cython and RPy? (I don't shell script a lot, so this may be a silly question) These days, I mostly seem to be able to get away with using single languages for applied machine learning, like a Python webserver that runs background machine learning tasks, or an Android app in Java that connects to a webserver written in Python. But back when I was doing less applied side stuff, I was similar to you in that things were spread across R and Python.
  - LukeShu 13y ago
    The Unix Shell is basically an IPC language. It's sole purpose is to be able to combine programs, and allow them to communicate. Instead of trying to make a package in one language work with another, it is a lot simpler to use Shell to connect together programs in any language.
  - chubot 13y ago
vdaniuk 13y ago
Should one learn R if one already has some experience in data analysis with Python?
- collyw 13y ago
  I would like to know the same. I work with a lot of bioinformaticians. The ones from a biology background (i.e. code quality is not usually important) seem to like R. The ones with a computer science background seem to go for Python, or Perl if they are old school. So far that has put me off R.
hadley 13y ago
I think R is actually a rather great first language, as the focus is usually on doing cool stuff, rather than developing a rigorous framework for programming. It's more important to put motivation before rigour, since if you focus only on rigour most people will lose interest before getting to the fun stuff.
- vikp 13y ago
  Great point. I still wish I coded in R more, just because the MTTC (mean time to cool) is so much lower. But, I feel like you can do the same with sklearn/ipython notebook/pandas if you ignore the parts of the "rigorous framework", and just focus on the syntax for interesting things.
oddthink 13y ago
I'm slowly moving my elaborate data-cleaning system from R to python. I never got the hang of R packages and ended up rolling my own "import" scheme, but now it all seems like I've passed some critical complexity point. I'm still doing all of my model estimation in R, and data.table + ggplot2 is still my go-to solution for interactive exploration and plotting, but I'm putting more and more of the process-csv, write-hdf5, remove out-of-bounds values, backfill, stratified sampling/bootstrap, etc., logic into python. I may end up moving more of the estimation itself into python, now that statsmodels and sklearn are more mature, but R does that pretty well. I've not tried Julia, but I have experimented with Java for some of the data-processing, but it's never seemed worth it, over python. Basic CSV processing is faster in python than in Java (although the python version uses much more CPU), so I haven't felt the need. (I'm probably limited more by disk bandwidth than CPU, but I've not done the tests required to prove that.) Julia seems nice, but it seems intent on copying all the questionable design choices of matlab, rather than using ideas from the numpy/kdb+/apl worlds.

EpiMath 13y ago

Interesting, thanks. ( Like your username too, despite him being slightly denigrated in the linked article. ) I personally think that it is good to have a basic grounding in some kind of fundamental "real" programming language before you get into s-plus ( R ) or any other more domain-specific language. The main reason being that you will better understand the compromises, limitations and shortcomings. I like R but as a programmer I've seen some awful code written in it!

RA_Fisher 13y ago
Yes, if you read further on the blog, you'll see I'm more than slightly denigrating. I read a book recently called The Cult of Statistical Significance and it's really turned me against Fisher. Not only do I see his methods as sub-standard to Bayesian analysis, he was really mean and disparaging himself to those folks. It's interesting, my username is what it is because he used to be my hero! Just 6 months ago, I really considered him to be the father of modern science. Now I basically see his work as mostly enabling mediocre scientists to rise through the ranks. This is what I mean by that: http://www.economist.com/blogs/graphicdetail/2013/10/daily-chart-2 http://www.economist.com/blogs/graphicdetail/2013/10/daily-c...
- EpiMath 13y ago
  I've spent my career thinking about these same issues. And certainly Fisher was, well, abrasive to put it mildly. I've read much of his early work and I think the intended context has mostly been lost. Many modern statisticians ( of the "frequentist" persuasion ) use a strange and awkward combination of fisherian and neyman-pearson. We talk about "p-values" but then interpret them as hypothesis tests with long-term error probabilities ( Fisher disavowed this interpretation of p-values, and insisted they did not have a long run probabilistic interpretation but were a measure only of evidence against the null in the particular experiment. ) I think Fisher gets a bad rap for a lot of later bastardization of his work. ( I'm sure it did not help that he was not a likable person, or his valid but misguided and testimony about tobacco. ) Still, I'm sympathetic to your position and can understand how you'd come to that way of thinking. I'm not convinced even a diehard Bayesian would completely disagree with Fisher's more restricted and stringent interpretation of p-values. Or at least they would see it as a big step up from the more common usage you are referencing.

elohesra 13y ago

Purely out of interest, what's so wrong with Windows as a development environment? I've seen this claim touted before on HN, but I've never seen an explanation (not doubting that there is one; I just haven't seen one). I develop on Windows every day, and I'm always happy to learn a new way to improve my workflow, so does Ubuntu (or Mac) actually make it easier to develop software, and if so then how? I'd also be interested to hear what kinds of software you've found it hard to develop on Windows.

Xophmeister 13y ago
In my (anecdotal) experience: A lot of tools (everything from editors to compilers, etc.) that are open source are from the Unix world. They're designed to work in a POSIX way and the ports to Windows (if they exist, or you're forced to use Cygwin) are a bit cludgy: You might just as well have used some kind of Unix in the first place. Windows dev tools are available, of course, but you often have to pay quite a lot for them. Good OSS exists for Windows, but it's a subset of what's readily available under nix, which doesn't encourage the hacker/hobbyist spirit.
- elohesra 13y ago
  Thanks for the reply. That does make sense. As a C# developer, I haven't really had call to use OSS IDEs, so I can't really comment, but I do remember having an ugly time trying to use an OSS interactive disassembler on Windows.
CraigJPerry 13y ago
>> what's so wrong with Windows as a development environment? Command line. When you're doing a task on a computer, some paradigms are better suited than others depending on the task. E.g. imagine delicately touching up pixels in a photo, guided by your artistic eye yet instructing the computer by laboriously typing many commands into a command line. There's no reason this couldn't work, it's just way more natural to use a pen & digtiser, or a mouse. When we're developing we're editing text and invoking commands. Editing text is covered by any major platform just fine, a lot of popular editors & IDEs are even cross platform. Invoking commands are sometimes covered by the IDE, sometimes the commands covered are pretty comprehensive (i'm thinking of Tom Christiansen's quip "Emacs is a nice operating system, but I prefer UNIX"). There's a limit to what tasks your IDE can cover, even though some do try really hard to cover all your needs (Eclipse has a web browser in it!) and when you hit that limit as we so often do in development, you need a command line. At that point, Windows kind of whistles while sheepishly looking off to one side. There's a related point about development dependencies, some libraries are hard and / or time consuming to build. A great package manager is a brilliant productivity booster. Mostly though, it's the point about the command line. (just to be clear, i have given powershell a good shot, i estimate i've authored almost 1k lines of it).