4 ms·

IMO the Unix philosophy of reuse is vital for data analysis. I recommend that people in this space learn to use the interactive shell and shell scripts well.

by chubot 13y ago

IMO the Unix philosophy of reuse is vital for data analysis. I recommend that people in this space learn to use the interactive shell and shell scripts well. I saw a comment that said "Shell is a REPL for C", which I think is quite pithy. People think too much about reusing R packages or Python packages. In my experience you can get things done a lot faster if you factor things into programs and not libraries. This lets you use multiple languages, and all real world analysis problems need multiple languages. (If you're only using one language, then you're likely only working on part of the problem). Right my toolset is Python, C++, and R, coordinated with shell scripts. I still like R for quick plotting, and of course data frames are essential, but I'm playing with Pandas now, which seems impressive. I don't think Python will ever catch up to R in terms of statistical functions, but in terms of plotting/munging/data frames it might. In the future there will be more languages, not less (Julia will add to the number of languages, not replace any). So being able to decompose a problem into separate, reasonably generic, programs is an important skill IMO. Most data analysis pipelines are a huge mess, but they don't have to be.

vikp 13y ago
That's an interesting way to go about it. Any reason for using shell scripts to coordinate the flow instead of things like Cython and RPy? (I don't shell script a lot, so this may be a silly question) These days, I mostly seem to be able to get away with using single languages for applied machine learning, like a Python webserver that runs background machine learning tasks, or an Android app in Java that connects to a webserver written in Python. But back when I was doing less applied side stuff, I was similar to you in that things were spread across R and Python.
- LukeShu 13y ago
  The Unix Shell is basically an IPC language. It's sole purpose is to be able to combine programs, and allow them to communicate. Instead of trying to make a package in one language work with another, it is a lot simpler to use Shell to connect together programs in any language.
- chubot 13y ago
  In my experience, RPy is annoying to get working, because you're dealing with Python versions and R versions together. It's brittle and usually unnecessary, as it's you can just serialize to CSV or JSON. If you can get away with using a single language that's good, and Python is probably the only one where that is possible (i.e. can write both prototype and production code, and data ingestion and machine learning). But I often have to use C++ because of the data size, and I think R's plotting is more convenient than anything in Python now. The shell scripts have their messiness and sharp edges, but they definitely save me many many lines of code. There is always some weird thing that needs to be integrated/automated and shell is almost always the right tool for that.
RA_Fisher 13y ago
This is an amazing suggestion. I hadn't thought of using shell scripts to modularize cross-language. Thank you.
- chubot 13y ago
  There have been some posts on Hacker News describing this philosophy (see below, can't find the HN comment links unfortunately). Another advantage is that it encourages separating your policy from mechanism. For example, you can make your sampling rate or number of buckets a command line flag, and have all your parameters in the shell scripts, rather than having various constants strewn about the code. And obviously you can use each language where it's stronger. Python is better (and much faster) for the data cleaning part, whereas you might want to use ggplot in R. http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html http://jeroenjanssens.com/2013/09/19/seven-command-line-tool... http://www.drbunsen.org/explorations-in-unix/ http://www.drbunsen.org/explorations-in-unix/ http://strata.oreilly.com/2011/04/data-hand-tools.html http://strata.oreilly.com/2011/04/data-hand-tools.html

by chubot 13y ago

vikp 13y ago

That's an interesting way to go about it. Any reason for using shell scripts to coordinate the flow instead of things like Cython and RPy? (I don't shell script a lot, so this may be a silly question) These days, I mostly seem to be able to get away with using single languages for applied machine learning, like a Python webserver that runs background machine learning tasks, or an Android app in Java that connects to a webserver written in Python. But back when I was doing less applied side stuff, I was similar to you in that things were spread across R and Python.

LukeShu 13y ago
The Unix Shell is basically an IPC language. It's sole purpose is to be able to combine programs, and allow them to communicate. Instead of trying to make a package in one language work with another, it is a lot simpler to use Shell to connect together programs in any language.
chubot 13y ago
In my experience, RPy is annoying to get working, because you're dealing with Python versions and R versions together. It's brittle and usually unnecessary, as it's you can just serialize to CSV or JSON. If you can get away with using a single language that's good, and Python is probably the only one where that is possible (i.e. can write both prototype and production code, and data ingestion and machine learning). But I often have to use C++ because of the data size, and I think R's plotting is more convenient than anything in Python now. The shell scripts have their messiness and sharp edges, but they definitely save me many many lines of code. There is always some weird thing that needs to be integrated/automated and shell is almost always the right tool for that.

RA_Fisher 13y ago

This is an amazing suggestion. I hadn't thought of using shell scripts to modularize cross-language. Thank you.

chubot 13y ago
There have been some posts on Hacker News describing this philosophy (see below, can't find the HN comment links unfortunately). Another advantage is that it encourages separating your policy from mechanism. For example, you can make your sampling rate or number of buckets a command line flag, and have all your parameters in the shell scripts, rather than having various constants strewn about the code. And obviously you can use each language where it's stronger. Python is better (and much faster) for the data cleaning part, whereas you might want to use ggplot in R. http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html http://jeroenjanssens.com/2013/09/19/seven-command-line-tool... http://www.drbunsen.org/explorations-in-unix/ http://www.drbunsen.org/explorations-in-unix/ http://strata.oreilly.com/2011/04/data-hand-tools.html http://strata.oreilly.com/2011/04/data-hand-tools.html