71 ms·

Useful Unix commands for exploring data

343 points by aks_c 12y ago

sheetjs 12y ago
caveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields: foo,"bar,baz" However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}' 3
- napD 12y ago
  Is there any workaround?
  - danielweber 12y ago
    Practically all Unix tools consider the comma-separated-but-you-can-use-quotes-to-override CSV file to be an abomination. [1] You have to have a crazy regexp to get around it. [1] Maybe someday they won't.
  - sheetjs 12y ago
    I primarily deal with Excel (xls) files nowadays. I wrote a command line tool to extract data: https://www.npmjs.org/package/j https://www.npmjs.org/package/j In my current workflow, I generate JSON from the excel files and use the really awesome JQ command (http://stedolan.github.io/jq/ http://stedolan.github.io/jq/) to process
  - hnha 12y ago
    use csvkit or something similar.
  - RBerenguel 12y ago
    csvfix is probably the best tool to deal with it. Csvfix, awk, sed are probably my "first line of data-attack". After that usually I can get to analysing, plotting or whatever I need to do.
  - jingo 12y ago
    Yes. Use lex/flex. You can write one-off (or reuseable) filters in minutes. lex/flex should be in every UNIX distribution that has a C compiler, but maybe that's changing.
  - mbreese 12y ago
    Don't use CSV files... If I'm working with a datafile where I expect the delimiter to be in one of the fields, there is something wrong. This is one reason why I always work with tab delimited files. Having an actual tab character isn't very common in free-text fields, at least in the data that I work with. Commas on the other hand, are quite common. Why one would select a field separator that was common in your data is beyond me (I know it's historical). Your data files might be different, in which case, maybe you should select a different field separator. Otherwise, no, there is no work around. If you have to quote fields, then you can't use the normal unix command line tools that tokenize fields.
  - dbro 12y ago
    yes! https://github.com/dbro/csvquote https://github.com/dbro/csvquote csvquote allows UNIX tools to work properly with quoted fields that contain delimiters inside the data. It is a simple translation tool that temporarily replaces the special characters occurring inside quotes with harmless non-printing characters. You do it as a first step in the pipeline, then do the regular operations using UNIX tools, and the last step of of the pipeline restores those troublesome characters back inside the data fields.
CraigJPerry 12y ago
>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There will be no work to do. There's a couple of options to deal with this, but the temporary intermediate file is my preference provided there's sufficient space - it's easily understood, if someone else comes across the construct in your script, they'll grok it.
- danielweber 12y ago
  I came here for that. I learned long ago, the hard way, to never ever use the same file for writing as reading. I was wondering if that rule had changed on me.
- aks_c 12y ago
  Thank you for inputs, how about this? uniq -u movies.csv > temp.csv temp.csv > movie.csv rm temp.csv
  - hnha 12y ago
    you could directly write to uniqMovie.csv in your example. I would do it like below but ONLY once I am certain it is exactly what I want. Usually I just make one clearly named result file per operation without touching the original. uniq -u movies.csv > /tmp/temp.csv && mv /temp/temp.csv movies.csv
  - icebraining 12y ago
    $ temp.csv > movie.csv temp.csv: command not found
    - kyllo 12y ago
      He forgot his cat.
    - deleted 12y ago
emeraldd 12y ago
uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.
- sheetjs 12y ago
  You can do this without sorting: awk '!x[$0]++'
  - jingo 12y ago
    I use something like this everyday: awk '!($0 in a);a[$0]; print}' I rarely if ever use uniq to remove duplicates. Sorting is expensive.
  - _delirium 12y ago
    That's usually faster where possible, but it may cause problems on large data sets, since it loads the entire set of unique strings (and their counts) into an in-memory hash table.
- danielweber 12y ago
  sort -u sort and uniq in one step.
  - barrkel 12y ago
    Indeed. uniq is usually only useful if you're also using -u, -d or -c.
- gav 12y ago
  > But that can screw with your header lines, so be careful there two. F=filename; (head -n 1 $F ; tail -n +2 $F | sort -u) | sponge $F To get counts of duplicates, you can use: sort filename | uniq -c | awk '$1 != 1'
  - smorrow 12y ago
peterwwillis 12y ago
You can also find tools designed for your dataset, like csvkit[1] , csvfix[2] , and other tools[3] (I even wrote my own CSV munging Unix tools in Perl back in the day) [1] http://csvkit.readthedocs.org/en/0.8.0/ http://csvkit.readthedocs.org/en/0.8.0/ [2] https://code.google.com/p/csvfix/ https://code.google.com/p/csvfix/ [3] https://unix.stackexchange.com/questions/7425/is-there-a-robust-command-line-tool-for-processing-csv-files https://unix.stackexchange.com/questions/7425/is-there-a-rob...
billyhoffman 12y ago
For last line, I always did tac [file] | head -n 1 Mainly because I can never remember basic sed commands (Strange, OS X doesn't seem to have tac, but Cygwin does...)
- petepete 12y ago
  If you installed coreutils with homebrew, it's called gtac.
- _delirium 12y ago
  The BSDish way of pronouncing 'tac' is 'tail -r'.
- alayne 12y ago
  Why not just tail -n [file]?
michaelmior 12y ago
It's good to note that `uniq -u` does remove duplicates, but it doesn't output any instances of a line which has been duplicated. This is probably not clear to a lot of people reading this.
- barrkel 12y ago
  `uniq` removes duplicates; `uniq -u` only shows unique lines.
  - michaelmior 12y ago
    Exactly. The point wasn't clear from reading the article.
nailer 12y ago
I love Unix pipelines, but chances are your data is structured in such a way that using regex based tools will break that structure unless you're very careful. You know that thing about not making HTML with regexs? Same rule applies to CSV, TSV, and XLSX. All these can be created, manipulated and read using Python, which is probably already on your system.
- icantthinkofone 12y ago
  iow, use Unix commands and pipes when you can. Don't use them when you can't.
- unhammer 12y ago
  There are unix tools that handle XML or CVS as well ;-) http://wiki.apertium.org/wiki/Xml_grep http://wiki.apertium.org/wiki/Xml_grep http://csvkit.readthedocs.org/en/latest/ http://csvkit.readthedocs.org/en/latest/
baldfat 12y ago
Certain people might miss the point of why to use command line. 1) I use this before using R or Python and ONLY do this when this is something I consistently need to be done all the time. Makes my R scripts shorter. 2) Somethings just need something simple to be fixed and these commands are just great. Learn awk and sed and your tools just go much larger in munging data.
- RBerenguel 12y ago
  Exactly! I had a longish period when I wanted to do everything with the same tool. Now, I try to pick the most efficient (for me, not the machine) to do it. Csvfix, awk, sed, jq and several other command line goodies make my life easier, the heavy lifting goes to R, gephi, or some ad-hoc Python, go or C
  - collyw 12y ago
    Fine if your only tool is Perl.
OneOneOneOne 12y ago
awk / gawk is super useful. For C/C++ programmers the language is very easy to learn. Try running "info gawk" for a very good guide. I've used gawk for many things ranging from data analysis to generate linker / loader code in an embedded build environment for a custom processor / sequencer. (You can even find a version to run from the Windows command prompt if you don't have Cygwin.)
hafabnew 12y ago
Not to sound too much like an Amazon product page, but if you like this, you'll probably quite like "Unix for Poets" - http://www.lsi.upc.edu/~padro/Unixforpoets.pdf http://www.lsi.upc.edu/~padro/Unixforpoets.pdf . It's my favourite 'intro' to text/data mangling using unix utils.
aabaker99 12y ago
You should mention this behavior of uniq (from the man page on my machine): Note: ’uniq’ does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use ‘sort -u’ without ‘uniq’. Your movies.csv file is already sorted, but you don't mention that sorting is important for using uniq, which may be misleading. $ cat tmp.txt AAAA AAAA BBBB DDDD BBBB $ uniq -d tmp.txt AAAA
zo1 12y ago
"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how your language works. Both things that will prevent you from being hindered next time around when you have to do a similar big-data job. Perhaps you are more comfortable using pre-defined lego-blocks to build your logic. Perhaps you understand the unix commands better than you do your chosen language. But understand that programming is the same, just in a different conceptual/knowledge space. And remember, always use the right tool for the job! (I use Unix commands daily as they're quick/dirty in a jiffy, but for complex tasks I am more productive solving the problem in a language I am comfortable in instead of searching through man pages for obscure flags/functionality)
- epistasis 12y ago
  Yeah, I think calling him the "unenlightened" one is pretty off base here. For performing the tasks outlined by his examples, Unix utilities are easier for the user as well as executing faster than writing your own code in a general purpose programming language, unless one puts in the time to tune the implementation. One could rebuild AWK in C and get similar performance, but why not just use some extremely simple AWK? And is anybody going to replicate the amazing speed of grep without a huge time investment? [1] There is a right tool for the job, and having seen dozens of programmers be exposed to new big data sets, I can tell you that the ones who become productive quickly and stay more productive are the ones who adopt whatever tool is best for the job, not the ones that stick to their one favorite programming language. In fact, a good sign of somebody who will quickly fail is someone who says "forget those Unix tools, I'm just going to write this in X". [1] http://ridiculousfish.com/blog/posts/old-age-and-treachery.html http://ridiculousfish.com/blog/posts/old-age-and-treachery.h...
  - derekp7 12y ago
    This is one area where I wish the Unix philosophy (reuse of tools) was taken a bit further. Too me, every command should be callable as a C library function. That way you wouldn't have to parse the human readable output through a pipe. Not only that, there needs to be both human-readable, as well as machine-readable output to all commands. For example I would love to be able to call "ps" from another script and easily select specific columns from an xml or json output.
platz 12y ago
http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/ http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...
pessimizer 12y ago
I'd like to repeat peterwwillis in saying that there are very Unixy tools that are designed for this, and update his link to my favorite, csvfix: http://neilb.bitbucket.org/csvfix/ http://neilb.bitbucket.org/csvfix/ Neat selling points: csvfix eval and csvfix exec also: the last commit to csvfix was 6 days ago; it's active, mature, and the developer is very responsive. If you can think of a capability that he hasn't yet, tell him and you'll have it in no time:)
lutusp 12y ago
Quote: "While dealing with big genetic data sets ..." What a great start. Unless he's a biologist, the author means generic, not genetic. The author goes on to show that he can use command-line utilities to accomplish what database clients do much more easily.
- mbreese 12y ago
  I think that he actually is a biologist. He refers to movies as a parallel universe. In which case, these tools are probably not all that helpful. Biological data is usually in the scale of either "Excel can handle it" (shudder) or "ginormous". In the later case, none of these would be all that useful, and CSV is not the standard format for most of the biological data that I see. Databases are less helpful than you'd imagine for this type of data as the schemas are not well defined. I am curious to know how JSON records would work for these data, because I could see something like that working for processing biological data files.
- kafkaesk 12y ago
  A few blog posts earlier, the author writes about "Network Analysis application in Genetic Studies ", so I am confident to say this isn't a typo. And for a quick-and-dirty custom analysis of big data sets, the unix tools might be a lot more convenient than databases.
- snowwrestler 12y ago
  He is a biologist.
CGamesPlay 12y ago
For working with complex CSV files, I highly recommend checking out CSVKit https://csvkit.readthedocs.org/en/0.8.0/ https://csvkit.readthedocs.org/en/0.8.0/ I've just started using it, and the only limitation I've so far encountered has been that there's no equivalent to awk (i.e. I want a way to evaluate a python expression on every line as part of a pipeline).
- vdm 12y ago
  Get words starting with "and" $ cat /usr/share/dict/words | py -fx 're.match(r"and", x)' | head -5 and andante andante's andantes andiron https://github.com/Russell91/pythonpy https://github.com/Russell91/pythonpy
  - CGamesPlay 12y ago
    Sorry, I meant: remove the characters "$" and "," from the 3rd column of a CSV file. Obviously the CSV file is quoted, since it has commas in the 3rd column, and so awk is no longer an acceptable solution.
jason_slack 12y ago
The author states: uniq -u movies.csv > temp.csv mv temp.csv movie.csv **Important thing to note here is uniq wont work if duplicate records are not adjacent. [Addition based on HN inputs] Would the fix here be to sort the lines first using the `sort` command first? Then `uniq`?
- icebraining 12y ago
  Yes, but not first, rather instead. "sort -u" both sorts and hides duplicates.
  - jason_slack 12y ago
    Thanks for the clarification!
  - mzs 12y ago
    except when you need uniq -c
- coderholic 12y ago
  Yes, and in fact you can just use sort's -u (unqiue) argument, and avoid uniq all together.
  - jason_slack 12y ago
    Thanks for the clarification!
zerop 12y ago
7 command-line tools for data science http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html http://jeroenjanssens.com/2013/09/19/seven-command-line-tool... Useful Unix commands for data science http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/ http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...
- jeroenjanssens 12y ago
  First blog post was the inspiration for a book, which is almost finished: http://datascienceatthecommandline.com http://datascienceatthecommandline.com
  - a3n 12y ago
    O'Reilly is having a 50% sale on all ebooks through 9 September. http://oreilly.com/ http://oreilly.com/ I just bought the early release of that exact book for $13.60, which was 60% off, because you get 60% off if you order $100 worth of prediscount ebooks. http://shop.oreilly.com/product/0636920032823.do http://shop.oreilly.com/product/0636920032823.do When the book is finished you get the final version. It's mostly already finished. "With Early Release ebooks, you get books in their earliest form — the author's raw and unedited content as he or she writes — so you can take advantage of these technologies long before the official release of these titles. You'll also receive updates when significant changes are made, new chapters as they're written, and the final ebook bundle."
WestCoastJustin 12y ago
My personal favorite is to use this pattern. You can do some extremely cool counts and group by operations at the command like [1]: grep '01/Jul/1995' NASA_access_log_Jul95 | awk '{print $1}' | sort | uniq -c | sort -h -r | head -n 15 Turns this: 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245 unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.0" 200 3985 199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] "GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0" 200 4085 burger.letters.com - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/countdown/liftoff.html HTTP/1.0" 304 0 199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0" 200 4179 Into this: 623 piweba3y.prodigy.com 547 piweba4y.prodigy.com 536 alyssa.prodigy.com 463 disarray.demon.co.uk 456 piweba1y.prodigy.com 417 www-b6.proxy.aol.com 350 burger.letters.com 300 poppy.hensa.ac.uk 279 www-b5.proxy.aol.com [1] https://sysadmincasts.com/episodes/28-cli-monday-cat-grep-awk-sort-and-uniq https://sysadmincasts.com/episodes/28-cli-monday-cat-grep-aw...
- vram22 12y ago
  I think there's something like that in the Kernighan and Pike book I referred to elsewhere in this thread, and also, that code looks similar to this technique: http://en.wikipedia.org/wiki/Decorate-sort-undecorate http://en.wikipedia.org/wiki/Decorate-sort-undecorate , i.e. Decorate-Sort-Undecorate (DSU), related to the Schwartzian transform.
- prlin 12y ago
  Similar ideas in Python with Generators and Co-routines: http://www.dabeaz.com/generators-uk/ http://www.dabeaz.com/generators-uk/
ngcazz 12y ago
No one gives a shit about cut. $ man 1 cut
- mprovost 12y ago
  I'm always surprised when people recommend awk for pulling delimited sections of lines out of a file, cut is so much easier to work with.
  - ciupicri 12y ago
    That's because cut sucks when fields can be separated by multiple space or tab characters. # printf '1 2\t3' | cut -f 2 3 # printf '1 2\t3' | awk '{print $2}' 2 # printf '1 2\t\t3' | cut -f 2 # printf '1 2\t\t3' | awk '{print $2}' 2
    - deleted 12y ago
      [deleted]
- vram22 12y ago
  paste and join are useful too; paste complements cut, and join is like the SQL join, but for text files.
forkandwait 12y ago
There is a command on freebsd for transposing text table rows to columns and vice versa, but I can't remember or find it now. It is in core, fwiw.
LiveTheDream 12y ago
I use this command very frequently to check how often an event occurs in a log file over time (specifically in 10-minute buckets), assuming the file is formatting like "INFO - [2014-08-27 16:16:29,578] Something something something" cat /path/to/logfile | grep PATTERN | sed 's/.*$2014-..-..$ $..$:$.$.*/\1 \2:\3x/' | uniq -c results in: 273 2014-08-27 14:5x 222 2014-08-27 15:0x 201 2014-08-27 15:1x 171 2014-08-27 15:2x 349 2014-08-27 15:3x 230 2014-08-27 15:4x 236 2014-08-27 15:5x 339 2014-08-27 16:0x 330 2014-08-27 16:1x This can subsequently be visualized with a tool like gnuplot or Excel.
- mprovost 12y ago
  Useless use of cat?
  - bch 12y ago
    "Don't pipe a cat" is how I'm used to describing what you're talking about -- it may have been a performance issue in days past, but these days I think it's simply a matter of style. Not that style is not important.
    - mprovost 12y ago
      This was drilled into me back in the usenet days. If you see a cat command with a single argument it's almost always replaceable by a shell redirection, or in this case just by passing the filename as an argument to grep. If you're processing lots of data like in the article there's no point in passing it through a separate command and pipe first.
      - smorrow 12y ago
        I think people like reading cat thefile | grepsedawk -opts 'prog' from left to right, and that they think the only alternative is grepsedawk -opts 'prog' thefile. But there's grep <thefile -opts 're'. I like that one best; it reads the same way you'd tend to think it.
      - 12y ago
xtacy 12y ago
I am surprised no one has mentioned datamash: http://www.gnu.org/software/datamash/ http://www.gnu.org/software/datamash/. It is a fantastic tool for doing quick filtering, group-by, aggregations, etc. Previous HN discussion: https://news.ycombinator.com/item?id=8130149 https://news.ycombinator.com/item?id=8130149
tdicola 12y ago
If you're on Windows, you owe it to yourself to check out a little known Microsoft utility called logparser: http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rocks-more-than-50-examples/ http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rock... It effectively lets you query a CSV (or many other log file formats/sources) with a SQL-like language. Very useful tool that I wish was available on Linux systems.
- teh_klev 12y ago
  LogParser is excellent, so many Windows admins have never heard of it which is a shame.
- 727374 12y ago
  LogParser is one of the few things I really miss from windows. I think there are unix equivalents, but I haven't had the time to invest in learning them. Pretty much every example in this article boiled down to 'Take this CSV and run a simple SQL query on it'. Yes you can do that by piping through various unix utilities or you could just use a tool mean specifically for the task. I'd like to see the article explore some more advanced cases, like rolling up a column. I actually had to do this yesterday and ended up opening my data in open office and using a pivot table.
- vdm 12y ago
  Nice find. You might also like http://matt.might.net/articles/sql-in-the-shell/ http://matt.might.net/articles/sql-in-the-shell/
- tstack 12y ago
  lnav (http://lnav.org http://lnav.org) provides SQL-queries-over-logs in the unix world. It's also a nice viewer for the logs themselves.
geh4y806 12y ago
just checking!
cyphunk 12y ago
really HN? if you find yourself depending heavily on the recommendations in this article you are doing data analysis wrong. Shell foo is relevant to data analysis only as much as regex is. In the same light depending on these methods too much is digging a deep knowledge ditch that in the end is going to limit and hinder you way more than the initial ingress time required to learn more capable data analytics frameworks or at least a scripting language. still, on international man page appreciate day this is a great reference. the only thing it is missing is gnuplot ascii graphs.
gesman 12y ago
Use splunk. 'nuff said.
known 12y ago
sort -T your_tmp_dir is very useful for sorting large data
known 12y ago
https://www.gnu.org/software/parallel/parallel_tutorial.html https://www.gnu.org/software/parallel/parallel_tutorial.html is very handy
dima55 12y ago
Then you can make plots by piping to https://github.com/dkogan/feedgnuplot https://github.com/dkogan/feedgnuplot
vesche 12y ago
Using basic Unix commands in trivial ways, am I missing something here?
- collyw 12y ago
  Its data science these days. Latest buzzword. Old is new.
letflow 12y ago
To run Unix commands on Terabytes of data, check out https://cloudbash.sh/ https://cloudbash.sh/. In addition to the standard Unix commands, their join, group-By operations are amazing. We guys are evaluating replacing our entire ETL with cloudbash!
squigs25 12y ago
sort before you uniq!
bmsherman 12y ago
I may as well plug my little program, which takes numbers read line-by-line in standard input and outputs a live-updating histogram (and some summary statistics) in the console! https://github.com/bmsherman/LiveHistogram https://github.com/bmsherman/LiveHistogram It's useful if you want to, say, get a quick feeling of the distribution of numbers in some column of text.
etrain 12y ago
Some more tips from someone who does this every day. 1) Be careful with CSV files and UNIX tools - most big CSV files with text fields have some subset of fields that are text quoted and character-escaped. This means that you might have "," in the middle of a string. Anything (like cut or awk) that depends on comma as a delimiter will not handle this situation well. 2) "cut" has shorter, easier to remember syntax than awk for selecting fields from a delimited file. 3) Did you know that you can do a database-style join directly in UNIX with common command line tools? See "join" - assumes your input files are sorted by join key. 4) As others have said - you almost invevitably want to run sort before you run uniq, since uniq only works on adjacent records. 5) sed doesn't get enough love: sed '1d' to delete the first line of a file. Useful for removing those pesky headers that interfere with later steps. Not to mention regex replacing, etc. 6) By the time you're doing most of this, you should probably be using python or R.
- ars 12y ago
  > you almost invevitably want to run sort before you run uniq And then you don't actually want uniq anyway since sort has a -u switch that removes duplicate lines.
  - potatosareok 12y ago
    What if you want uniq -c? Any simple way to replicate that functionality better then...sort | uniq -c?
    - ars 12y ago
      Then you run uniq -c (which I do all the time). But for the examples in the main article sort -u would be fine.
- collyw 12y ago
  Actually I would say Perl is more appropriate. I went back to Perl after 4 years for this sort of task, as it has so many features built into the syntax. Plus it can be run as a one liner.
forkandwait 12y ago
"rs" for "reshape array". Found only on FreeBSD systems (yes, we are better... smile) For example, transpose a text file: ~/ (j=0,r=1)$ cat foo.txt a b c d e f ~/ (j=0,r=0)$ cat foo.txt | rs -T a d b e c f Honestly I have never used in production, but I still think it is way cool. Also, being forced to work in a non-Unix environment, I am always reminded how much I wish everything were either text files, zipped text files, or a SQL database. I know for really big data (bigger than our typical 10^7 row dataset, like imagery or genetics), you have to expand into things like HDF5, but part of my first data cleaning sequence is often to take something out of Excel or whatever and make a text file from it and apply unix tools.
- jingo 12y ago
  "Found only on FreeBSD..." Also found on NetBSD, OpenBSD and DragonFlyBSD.
  - clarry 12y ago
    Utree tells rs dates back to early 80s... http://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/new/new/tools/man/rs.1 http://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/...
- super_mario 12y ago
  It's also on Mac OS X by default :D.
CharlesMerriam2 12y ago
Do you have a pastebin of the CSV file? Time to play...
zo1 12y ago
Just one other thing I'd like to mention before everyone moves on to another topic. Not all of the unix commands are equal, and some have features that others don't. E.g. I mainly work on AIX, and a lot of the commands are simply not the same as what they are on more standard linux flavors. From what I've heard, this applies between different distros as well. Not so much the case with standard programming languages that are portable. E.g. Python. Unless you take in to account Jython, etc.
jweslley 12y ago
I built stats-tools for use in place of awk for basic statistics. https://github.com/jweslley/stats-tools https://github.com/jweslley/stats-tools

343 points by aks_c 12y ago

caveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields: foo,"bar,baz" However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}' 3

napD 12y ago
Is there any workaround?
- danielweber 12y ago
  Practically all Unix tools consider the comma-separated-but-you-can-use-quotes-to-override CSV file to be an abomination. [1] You have to have a crazy regexp to get around it. [1] Maybe someday they won't.
- sheetjs 12y ago
  I primarily deal with Excel (xls) files nowadays. I wrote a command line tool to extract data: https://www.npmjs.org/package/j https://www.npmjs.org/package/j In my current workflow, I generate JSON from the excel files and use the really awesome JQ command (http://stedolan.github.io/jq/ http://stedolan.github.io/jq/) to process
- hnha 12y ago
  use csvkit or something similar.
- RBerenguel 12y ago
  csvfix is probably the best tool to deal with it. Csvfix, awk, sed are probably my "first line of data-attack". After that usually I can get to analysing, plotting or whatever I need to do.
- jingo 12y ago
  Yes. Use lex/flex. You can write one-off (or reuseable) filters in minutes. lex/flex should be in every UNIX distribution that has a C compiler, but maybe that's changing.
- mbreese 12y ago
  Don't use CSV files... If I'm working with a datafile where I expect the delimiter to be in one of the fields, there is something wrong. This is one reason why I always work with tab delimited files. Having an actual tab character isn't very common in free-text fields, at least in the data that I work with. Commas on the other hand, are quite common. Why one would select a field separator that was common in your data is beyond me (I know it's historical). Your data files might be different, in which case, maybe you should select a different field separator. Otherwise, no, there is no work around. If you have to quote fields, then you can't use the normal unix command line tools that tokenize fields.
- dbro 12y ago
  yes! https://github.com/dbro/csvquote https://github.com/dbro/csvquote csvquote allows UNIX tools to work properly with quoted fields that contain delimiters inside the data. It is a simple translation tool that temporarily replaces the special characters occurring inside quotes with harmless non-printing characters. You do it as a first step in the pipeline, then do the regular operations using UNIX tools, and the last step of of the pipeline restores those troublesome characters back inside the data fields.

CraigJPerry 12y ago

>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There will be no work to do. There's a couple of options to deal with this, but the temporary intermediate file is my preference provided there's sufficient space - it's easily understood, if someone else comes across the construct in your script, they'll grok it.

danielweber 12y ago
I came here for that. I learned long ago, the hard way, to never ever use the same file for writing as reading. I was wondering if that rule had changed on me.
aks_c 12y ago
Thank you for inputs, how about this? uniq -u movies.csv > temp.csv temp.csv > movie.csv rm temp.csv
- hnha 12y ago
  you could directly write to uniqMovie.csv in your example. I would do it like below but ONLY once I am certain it is exactly what I want. Usually I just make one clearly named result file per operation without touching the original. uniq -u movies.csv > /tmp/temp.csv && mv /temp/temp.csv movies.csv
- icebraining 12y ago
  $ temp.csv > movie.csv temp.csv: command not found
  - kyllo 12y ago
    He forgot his cat.
  - deleted 12y ago

emeraldd 12y ago

uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.

sheetjs 12y ago
You can do this without sorting: awk '!x[$0]++'
- jingo 12y ago
  I use something like this everyday: awk '!($0 in a);a[$0]; print}' I rarely if ever use uniq to remove duplicates. Sorting is expensive.
- _delirium 12y ago
  That's usually faster where possible, but it may cause problems on large data sets, since it loads the entire set of unique strings (and their counts) into an in-memory hash table.
danielweber 12y ago
sort -u sort and uniq in one step.
- barrkel 12y ago
  Indeed. uniq is usually only useful if you're also using -u, -d or -c.
gav 12y ago
> But that can screw with your header lines, so be careful there two. F=filename; (head -n 1 $F ; tail -n +2 $F | sort -u) | sponge $F To get counts of duplicates, you can use: sort filename | uniq -c | awk '$1 != 1'
- smorrow 12y ago

peterwwillis 12y ago

You can also find tools designed for your dataset, like csvkit[1] , csvfix[2] , and other tools[3] (I even wrote my own CSV munging Unix tools in Perl back in the day) [1] http://csvkit.readthedocs.org/en/0.8.0/ http://csvkit.readthedocs.org/en/0.8.0/ [2] https://code.google.com/p/csvfix/ https://code.google.com/p/csvfix/ [3] https://unix.stackexchange.com/questions/7425/is-there-a-robust-command-line-tool-for-processing-csv-files https://unix.stackexchange.com/questions/7425/is-there-a-rob...

billyhoffman 12y ago

For last line, I always did tac [file] | head -n 1 Mainly because I can never remember basic sed commands (Strange, OS X doesn't seem to have tac, but Cygwin does...)

petepete 12y ago
If you installed coreutils with homebrew, it's called gtac.
_delirium 12y ago
The BSDish way of pronouncing 'tac' is 'tail -r'.
alayne 12y ago
Why not just tail -n [file]?

michaelmior 12y ago

It's good to note that `uniq -u` does remove duplicates, but it doesn't output any instances of a line which has been duplicated. This is probably not clear to a lot of people reading this.

barrkel 12y ago
`uniq` removes duplicates; `uniq -u` only shows unique lines.
- michaelmior 12y ago
  Exactly. The point wasn't clear from reading the article.

nailer 12y ago

I love Unix pipelines, but chances are your data is structured in such a way that using regex based tools will break that structure unless you're very careful. You know that thing about not making HTML with regexs? Same rule applies to CSV, TSV, and XLSX. All these can be created, manipulated and read using Python, which is probably already on your system.

icantthinkofone 12y ago
iow, use Unix commands and pipes when you can. Don't use them when you can't.
unhammer 12y ago
There are unix tools that handle XML or CVS as well ;-) http://wiki.apertium.org/wiki/Xml_grep http://wiki.apertium.org/wiki/Xml_grep http://csvkit.readthedocs.org/en/latest/ http://csvkit.readthedocs.org/en/latest/

baldfat 12y ago

Certain people might miss the point of why to use command line. 1) I use this before using R or Python and ONLY do this when this is something I consistently need to be done all the time. Makes my R scripts shorter. 2) Somethings just need something simple to be fixed and these commands are just great. Learn awk and sed and your tools just go much larger in munging data.

RBerenguel 12y ago
Exactly! I had a longish period when I wanted to do everything with the same tool. Now, I try to pick the most efficient (for me, not the machine) to do it. Csvfix, awk, sed, jq and several other command line goodies make my life easier, the heavy lifting goes to R, gephi, or some ad-hoc Python, go or C
- collyw 12y ago
  Fine if your only tool is Perl.

OneOneOneOne 12y ago

awk / gawk is super useful. For C/C++ programmers the language is very easy to learn. Try running "info gawk" for a very good guide. I've used gawk for many things ranging from data analysis to generate linker / loader code in an embedded build environment for a custom processor / sequencer. (You can even find a version to run from the Windows command prompt if you don't have Cygwin.)

hafabnew 12y ago

Not to sound too much like an Amazon product page, but if you like this, you'll probably quite like "Unix for Poets" - http://www.lsi.upc.edu/~padro/Unixforpoets.pdf http://www.lsi.upc.edu/~padro/Unixforpoets.pdf . It's my favourite 'intro' to text/data mangling using unix utils.

aabaker99 12y ago

You should mention this behavior of uniq (from the man page on my machine): Note: ’uniq’ does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use ‘sort -u’ without ‘uniq’. Your movies.csv file is already sorted, but you don't mention that sorting is important for using uniq, which may be misleading. $ cat tmp.txt AAAA AAAA BBBB DDDD BBBB $ uniq -d tmp.txt AAAA

zo1 12y ago

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how your language works. Both things that will prevent you from being hindered next time around when you have to do a similar big-data job. Perhaps you are more comfortable using pre-defined lego-blocks to build your logic. Perhaps you understand the unix commands better than you do your chosen language. But understand that programming is the same, just in a different conceptual/knowledge space. And remember, always use the right tool for the job! (I use Unix commands daily as they're quick/dirty in a jiffy, but for complex tasks I am more productive solving the problem in a language I am comfortable in instead of searching through man pages for obscure flags/functionality)

epistasis 12y ago
Yeah, I think calling him the "unenlightened" one is pretty off base here. For performing the tasks outlined by his examples, Unix utilities are easier for the user as well as executing faster than writing your own code in a general purpose programming language, unless one puts in the time to tune the implementation. One could rebuild AWK in C and get similar performance, but why not just use some extremely simple AWK? And is anybody going to replicate the amazing speed of grep without a huge time investment? [1] There is a right tool for the job, and having seen dozens of programmers be exposed to new big data sets, I can tell you that the ones who become productive quickly and stay more productive are the ones who adopt whatever tool is best for the job, not the ones that stick to their one favorite programming language. In fact, a good sign of somebody who will quickly fail is someone who says "forget those Unix tools, I'm just going to write this in X". [1] http://ridiculousfish.com/blog/posts/old-age-and-treachery.html http://ridiculousfish.com/blog/posts/old-age-and-treachery.h...
- derekp7 12y ago
  This is one area where I wish the Unix philosophy (reuse of tools) was taken a bit further. Too me, every command should be callable as a C library function. That way you wouldn't have to parse the human readable output through a pipe. Not only that, there needs to be both human-readable, as well as machine-readable output to all commands. For example I would love to be able to call "ps" from another script and easily select specific columns from an xml or json output.

platz 12y ago

http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/ http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...

pessimizer 12y ago

I'd like to repeat peterwwillis in saying that there are very Unixy tools that are designed for this, and update his link to my favorite, csvfix: http://neilb.bitbucket.org/csvfix/ http://neilb.bitbucket.org/csvfix/ Neat selling points: csvfix eval and csvfix exec also: the last commit to csvfix was 6 days ago; it's active, mature, and the developer is very responsive. If you can think of a capability that he hasn't yet, tell him and you'll have it in no time:)

lutusp 12y ago

Quote: "While dealing with big genetic data sets ..." What a great start. Unless he's a biologist, the author means generic, not genetic. The author goes on to show that he can use command-line utilities to accomplish what database clients do much more easily.

mbreese 12y ago
I think that he actually is a biologist. He refers to movies as a parallel universe. In which case, these tools are probably not all that helpful. Biological data is usually in the scale of either "Excel can handle it" (shudder) or "ginormous". In the later case, none of these would be all that useful, and CSV is not the standard format for most of the biological data that I see. Databases are less helpful than you'd imagine for this type of data as the schemas are not well defined. I am curious to know how JSON records would work for these data, because I could see something like that working for processing biological data files.
kafkaesk 12y ago
A few blog posts earlier, the author writes about "Network Analysis application in Genetic Studies ", so I am confident to say this isn't a typo. And for a quick-and-dirty custom analysis of big data sets, the unix tools might be a lot more convenient than databases.
snowwrestler 12y ago
He is a biologist.

CGamesPlay 12y ago

For working with complex CSV files, I highly recommend checking out CSVKit https://csvkit.readthedocs.org/en/0.8.0/ https://csvkit.readthedocs.org/en/0.8.0/ I've just started using it, and the only limitation I've so far encountered has been that there's no equivalent to awk (i.e. I want a way to evaluate a python expression on every line as part of a pipeline).

vdm 12y ago
Get words starting with "and" $ cat /usr/share/dict/words | py -fx 're.match(r"and", x)' | head -5 and andante andante's andantes andiron https://github.com/Russell91/pythonpy https://github.com/Russell91/pythonpy
- CGamesPlay 12y ago
  Sorry, I meant: remove the characters "$" and "," from the 3rd column of a CSV file. Obviously the CSV file is quoted, since it has commas in the 3rd column, and so awk is no longer an acceptable solution.

jason_slack 12y ago

The author states: uniq -u movies.csv > temp.csv mv temp.csv movie.csv **Important thing to note here is uniq wont work if duplicate records are not adjacent. [Addition based on HN inputs] Would the fix here be to sort the lines first using the `sort` command first? Then `uniq`?

icebraining 12y ago
Yes, but not first, rather instead. "sort -u" both sorts and hides duplicates.
- jason_slack 12y ago
  Thanks for the clarification!
- mzs 12y ago
  except when you need uniq -c
coderholic 12y ago
Yes, and in fact you can just use sort's -u (unqiue) argument, and avoid uniq all together.
- jason_slack 12y ago
  Thanks for the clarification!

zerop 12y ago

7 command-line tools for data science http://jeroenjanssens.com/2013/09/19/seven-command-line-tools-for-data-science.html http://jeroenjanssens.com/2013/09/19/seven-command-line-tool... Useful Unix commands for data science http://www.gregreda.com/2013/07/15/unix-commands-for-data-science/ http://www.gregreda.com/2013/07/15/unix-commands-for-data-sc...

jeroenjanssens 12y ago
First blog post was the inspiration for a book, which is almost finished: http://datascienceatthecommandline.com http://datascienceatthecommandline.com
- a3n 12y ago
  O'Reilly is having a 50% sale on all ebooks through 9 September. http://oreilly.com/ http://oreilly.com/ I just bought the early release of that exact book for $13.60, which was 60% off, because you get 60% off if you order $100 worth of prediscount ebooks. http://shop.oreilly.com/product/0636920032823.do http://shop.oreilly.com/product/0636920032823.do When the book is finished you get the final version. It's mostly already finished. "With Early Release ebooks, you get books in their earliest form — the author's raw and unedited content as he or she writes — so you can take advantage of these technologies long before the official release of these titles. You'll also receive updates when significant changes are made, new chapters as they're written, and the final ebook bundle."

WestCoastJustin 12y ago

My personal favorite is to use this pattern. You can do some extremely cool counts and group by operations at the command like [1]: grep '01/Jul/1995' NASA_access_log_Jul95 | awk '{print $1}' | sort | uniq -c | sort -h -r | head -n 15 Turns this: 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245 unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.0" 200 3985 199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] "GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0" 200 4085 burger.letters.com - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/countdown/liftoff.html HTTP/1.0" 304 0 199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0" 200 4179 Into this: 623 piweba3y.prodigy.com 547 piweba4y.prodigy.com 536 alyssa.prodigy.com 463 disarray.demon.co.uk 456 piweba1y.prodigy.com 417 www-b6.proxy.aol.com 350 burger.letters.com 300 poppy.hensa.ac.uk 279 www-b5.proxy.aol.com [1] https://sysadmincasts.com/episodes/28-cli-monday-cat-grep-awk-sort-and-uniq https://sysadmincasts.com/episodes/28-cli-monday-cat-grep-aw...

vram22 12y ago
I think there's something like that in the Kernighan and Pike book I referred to elsewhere in this thread, and also, that code looks similar to this technique: http://en.wikipedia.org/wiki/Decorate-sort-undecorate http://en.wikipedia.org/wiki/Decorate-sort-undecorate , i.e. Decorate-Sort-Undecorate (DSU), related to the Schwartzian transform.
prlin 12y ago
Similar ideas in Python with Generators and Co-routines: http://www.dabeaz.com/generators-uk/ http://www.dabeaz.com/generators-uk/

ngcazz 12y ago

No one gives a shit about cut. $ man 1 cut

mprovost 12y ago
I'm always surprised when people recommend awk for pulling delimited sections of lines out of a file, cut is so much easier to work with.
- ciupicri 12y ago
  That's because cut sucks when fields can be separated by multiple space or tab characters. # printf '1 2\t3' | cut -f 2 3 # printf '1 2\t3' | awk '{print $2}' 2 # printf '1 2\t\t3' | cut -f 2 # printf '1 2\t\t3' | awk '{print $2}' 2
  - deleted 12y ago
    [deleted]
vram22 12y ago
paste and join are useful too; paste complements cut, and join is like the SQL join, but for text files.

forkandwait 12y ago

There is a command on freebsd for transposing text table rows to columns and vice versa, but I can't remember or find it now. It is in core, fwiw.

LiveTheDream 12y ago

I use this command very frequently to check how often an event occurs in a log file over time (specifically in 10-minute buckets), assuming the file is formatting like "INFO - [2014-08-27 16:16:29,578] Something something something" cat /path/to/logfile | grep PATTERN | sed 's/.*$2014-..-..$ $..$:$.$.*/\1 \2:\3x/' | uniq -c results in: 273 2014-08-27 14:5x 222 2014-08-27 15:0x 201 2014-08-27 15:1x 171 2014-08-27 15:2x 349 2014-08-27 15:3x 230 2014-08-27 15:4x 236 2014-08-27 15:5x 339 2014-08-27 16:0x 330 2014-08-27 16:1x This can subsequently be visualized with a tool like gnuplot or Excel.

mprovost 12y ago
Useless use of cat?
- bch 12y ago
  "Don't pipe a cat" is how I'm used to describing what you're talking about -- it may have been a performance issue in days past, but these days I think it's simply a matter of style. Not that style is not important.
  - mprovost 12y ago
    This was drilled into me back in the usenet days. If you see a cat command with a single argument it's almost always replaceable by a shell redirection, or in this case just by passing the filename as an argument to grep. If you're processing lots of data like in the article there's no point in passing it through a separate command and pipe first.
    - smorrow 12y ago
      I think people like reading cat thefile | grepsedawk -opts 'prog' from left to right, and that they think the only alternative is grepsedawk -opts 'prog' thefile. But there's grep <thefile -opts 're'. I like that one best; it reads the same way you'd tend to think it.
    - 12y ago

xtacy 12y ago

I am surprised no one has mentioned datamash: http://www.gnu.org/software/datamash/ http://www.gnu.org/software/datamash/. It is a fantastic tool for doing quick filtering, group-by, aggregations, etc. Previous HN discussion: https://news.ycombinator.com/item?id=8130149 https://news.ycombinator.com/item?id=8130149

tdicola 12y ago

If you're on Windows, you owe it to yourself to check out a little known Microsoft utility called logparser: http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rocks-more-than-50-examples/ http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rock... It effectively lets you query a CSV (or many other log file formats/sources) with a SQL-like language. Very useful tool that I wish was available on Linux systems.

teh_klev 12y ago
LogParser is excellent, so many Windows admins have never heard of it which is a shame.
727374 12y ago
LogParser is one of the few things I really miss from windows. I think there are unix equivalents, but I haven't had the time to invest in learning them. Pretty much every example in this article boiled down to 'Take this CSV and run a simple SQL query on it'. Yes you can do that by piping through various unix utilities or you could just use a tool mean specifically for the task. I'd like to see the article explore some more advanced cases, like rolling up a column. I actually had to do this yesterday and ended up opening my data in open office and using a pivot table.
vdm 12y ago
Nice find. You might also like http://matt.might.net/articles/sql-in-the-shell/ http://matt.might.net/articles/sql-in-the-shell/
tstack 12y ago
lnav (http://lnav.org http://lnav.org) provides SQL-queries-over-logs in the unix world. It's also a nice viewer for the logs themselves.

geh4y806 12y ago

just checking!

cyphunk 12y ago

really HN? if you find yourself depending heavily on the recommendations in this article you are doing data analysis wrong. Shell foo is relevant to data analysis only as much as regex is. In the same light depending on these methods too much is digging a deep knowledge ditch that in the end is going to limit and hinder you way more than the initial ingress time required to learn more capable data analytics frameworks or at least a scripting language. still, on international man page appreciate day this is a great reference. the only thing it is missing is gnuplot ascii graphs.

gesman 12y ago

Use splunk. 'nuff said.

known 12y ago

sort -T your_tmp_dir is very useful for sorting large data

known 12y ago

https://www.gnu.org/software/parallel/parallel_tutorial.html https://www.gnu.org/software/parallel/parallel_tutorial.html is very handy

dima55 12y ago

Then you can make plots by piping to https://github.com/dkogan/feedgnuplot https://github.com/dkogan/feedgnuplot

vesche 12y ago

Using basic Unix commands in trivial ways, am I missing something here?

collyw 12y ago
Its data science these days. Latest buzzword. Old is new.

letflow 12y ago

To run Unix commands on Terabytes of data, check out https://cloudbash.sh/ https://cloudbash.sh/. In addition to the standard Unix commands, their join, group-By operations are amazing. We guys are evaluating replacing our entire ETL with cloudbash!

squigs25 12y ago

sort before you uniq!

bmsherman 12y ago

I may as well plug my little program, which takes numbers read line-by-line in standard input and outputs a live-updating histogram (and some summary statistics) in the console! https://github.com/bmsherman/LiveHistogram https://github.com/bmsherman/LiveHistogram It's useful if you want to, say, get a quick feeling of the distribution of numbers in some column of text.

etrain 12y ago

Some more tips from someone who does this every day. 1) Be careful with CSV files and UNIX tools - most big CSV files with text fields have some subset of fields that are text quoted and character-escaped. This means that you might have "," in the middle of a string. Anything (like cut or awk) that depends on comma as a delimiter will not handle this situation well. 2) "cut" has shorter, easier to remember syntax than awk for selecting fields from a delimited file. 3) Did you know that you can do a database-style join directly in UNIX with common command line tools? See "join" - assumes your input files are sorted by join key. 4) As others have said - you almost invevitably want to run sort before you run uniq, since uniq only works on adjacent records. 5) sed doesn't get enough love: sed '1d' to delete the first line of a file. Useful for removing those pesky headers that interfere with later steps. Not to mention regex replacing, etc. 6) By the time you're doing most of this, you should probably be using python or R.

ars 12y ago
> you almost invevitably want to run sort before you run uniq And then you don't actually want uniq anyway since sort has a -u switch that removes duplicate lines.
- potatosareok 12y ago
  What if you want uniq -c? Any simple way to replicate that functionality better then...sort | uniq -c?
  - ars 12y ago
    Then you run uniq -c (which I do all the time). But for the examples in the main article sort -u would be fine.
collyw 12y ago
Actually I would say Perl is more appropriate. I went back to Perl after 4 years for this sort of task, as it has so many features built into the syntax. Plus it can be run as a one liner.

forkandwait 12y ago

"rs" for "reshape array". Found only on FreeBSD systems (yes, we are better... smile) For example, transpose a text file: ~/ (j=0,r=1)$ cat foo.txt a b c d e f ~/ (j=0,r=0)$ cat foo.txt | rs -T a d b e c f Honestly I have never used in production, but I still think it is way cool. Also, being forced to work in a non-Unix environment, I am always reminded how much I wish everything were either text files, zipped text files, or a SQL database. I know for really big data (bigger than our typical 10^7 row dataset, like imagery or genetics), you have to expand into things like HDF5, but part of my first data cleaning sequence is often to take something out of Excel or whatever and make a text file from it and apply unix tools.

jingo 12y ago
"Found only on FreeBSD..." Also found on NetBSD, OpenBSD and DragonFlyBSD.
- clarry 12y ago
  Utree tells rs dates back to early 80s... http://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/new/new/tools/man/rs.1 http://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/...
super_mario 12y ago
It's also on Mac OS X by default :D.

CharlesMerriam2 12y ago

Do you have a pastebin of the CSV file? Time to play...

zo1 12y ago

Just one other thing I'd like to mention before everyone moves on to another topic. Not all of the unix commands are equal, and some have features that others don't. E.g. I mainly work on AIX, and a lot of the commands are simply not the same as what they are on more standard linux flavors. From what I've heard, this applies between different distros as well. Not so much the case with standard programming languages that are portable. E.g. Python. Unless you take in to account Jython, etc.

jweslley 12y ago

I built stats-tools for use in place of awk for basic statistics. https://github.com/jweslley/stats-tools https://github.com/jweslley/stats-tools