7 ms·
I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine
by bloopernova 7y ago
I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into.
I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job.
For instance, "mail server x is getting '81126 delayed delivery' from google messages in the logs, find out who is sending those messages".
# get all the lines with the 81126 message. Get the queue IDs, exclude duplicates, save them in a file.
cat maillog.txt | grep 81126 | awk '{print $6}' | sort | uniq | cut -d':' -f1 > queue-ids.txt
# Grep for entries in that file, get the from addresses, exclude duplicates.
cat maillog.txt | grep -F -f queue-ids.txt | grep 'from=<' | awk '{print $7}' | cut -d'<' -f2 | cut -d'>' -f1 | sort | uniq
Each of those 2 one-liners was built up pipe-by-pipe, looking at the output, finding what I needed. It's not pretty, it's not elegant, but it works. I'm sure there's a million ways that a thousand different languages could do this more elegantly, but it's what I know, and it works for me.
- darau1 7y ago> it's not elegant I completely disagree.
- samstave 7y agoEloquently put!
- bloopernova 7y agoThank you, I too often talk down what I do.
- rhombocombus 7y agoIts ubiquity and performance open up all kinds of sophisticated data processing on a huge variety of *nix implementations. Whether it's one liners or giant data scrubs, awk is a tool that you can almost always count on having access to, even in the most restrictive or arcane environments.
- jcims 7y agoI know you’re not asking for awk protips but you can prefix the block with a match condition for processing. ... | grep foo | awk ‘{print $6}’ | ... becomes ... | awk ‘/foo/{print $6}’ | ... If you start working this into your awk habits you’ll find delightful little edge cases that you can handle with other expressions before the block (you can, for example, match specific fields).
- bloopernova 7y agoI always forget about that, and I should try more to remember it. Thank you for the tip!
- mistahenry 7y agoOn the other hand, grep can be far faster for searching alone than awk. I almost always use an initial grep for the string that will most reduce the input to the rest of the pipeline. Later, it feels idiomatic to mix in awk with matches like you suggested
- davidgould 7y agoDepends on the awk. mawk is surprisingly fast.
- just_myles 7y agoRight. I don't consider that particular exhaustive at all and this has helped me when I wanted to do quick searches.
- 3xblah 7y agoNo one has mentioned changing the default field separator, e.g., awk FS=: '{print $1}' instead of cut -d: -f1 awk FS="<" '{print $2}' instead of cut -d'<' -f2 awk FS=">" '{print $1}' instead of cut -d'>' -f1
- chaps 7y agoNo need to explicitly set FS! Just use: echo test,123 | awk -F, '{print $1}'
- jimbokun 7y agoIt's far more elegant and concise than any other scripting language I can think of using to accomplish the same thing. As the article points out, other languages will have a lot more ceremony around opening and closing the file, breaking the input into fields, initializing variables, etc.
- 3fe9a03ccd14ca5 7y agoAs part of a practical component to any software engineering degree should be a simple course on common Unix tools, covering grep, awk, sed, PCRE, and git. A little bit of knowledge here goes a LONG way.
- bloopernova 7y agoI wholeheartedly agree. I've seen people agonize for days over results from Splunk that they want to turn into something more user-friendly. 15 minutes of messing around with the basic command line Unix tools has that information in a perfect format for their needs. This is something I need to bring up to my coworkers, I should write some sort of basic guide to unix tools for them.
- nextos 7y agoI disagree, it's quite elegant if you think in terms of relational algebra operators: * Projection (Π): awk and cut for simple cases * Selection (σ): grep for simple cases, otherwise sed & awk * Rename (ρ): sed * Set operators: join, comm...
- agumonkey 7y agontfsql dreams
- chongli 7y agoBravo! This is one of the most insightful comments I've read in a long time! I have been using some of these tools for years but I never thought of describing them this way. Now I can think of writing a complex query in relational algebra and translating it into these commands in a very natural way.
- _TwoFinger 7y agoHere's an interesting article that links shell scripting and relational algebra - http://matt.might.net/articles/sql-in-the-shell/ http://matt.might.net/articles/sql-in-the-shell/
- graphpapa 7y agoI’m pretty mathsy but I don’t get this
- rz2k 7y agoIt is from relational algebra used in database theory. There is an excerpt from one of the first MOOCS offered here on Lagunitas now.[1] It is pretty intuitive once you get the hang of it. [1] https://lagunita.stanford.edu/courses/DB/RA/SelfPaced/course/ https://lagunita.stanford.edu/courses/DB/RA/SelfPaced/course...
- graphpapa 7y agoThank you for this context