13 ms·

Awk in 20 Minutes

504 points by craigkerstiens 12y ago

carapace 12y ago
The second comma in the second sentence has got to go. "on files, usually structured" <-- that one. There are some other ones too. Other than that this is really great!
hayksaakian 12y ago
someone should add this to http://learnxinyminutes.com/ http://learnxinyminutes.com/
decisiveness 12y ago
This is probably the most concise introduction to awk I've read. For more extensions[0] and advanced features like arrays of arrays and array sorting, there's also gawk. And for larger files there's performance driven mawk which can drastically increase processing speed[1]. [0]https://www.gnu.org/software/gawk/manual/html_node/Extension-Samples.html https://www.gnu.org/software/gawk/manual/html_node/Extension... [1]http://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ http://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-a...
na85 12y ago
Story time. Back in ~2005 when I was still very new to Linux I had this old Dell that I put Gentoo on. It was a 900MHz Intel Coppermine with a minuscule amount of memory, so things compiled very slow. Being the wise Linux guru that I was back then, I decided that I would emerge (compile) all of X, fluxbox, OpenOffice, and probably 1 or 2 other things. I didn't have distcc set up. It compiled for days. One thing I really did enjoy was watching the cryptic messages crawl by, and one that always mystified me was during the configure stages: checking for gawk... gawk It seemed like a total nonsense word. To this day I still have fond memories every time someone mentions (g)awk.
wazoox 12y ago
I use perl one-liners, thanks to the power of -n, -i and -p switches. perl -n -e <expression> <file> runs the expression on each and every line of <file> (can be STDIN of course). perl -p -e <expression> <file> prints out every line for which the expression returns true. The -i option allows treating files in place. Add -i<extension> to create a <file>.<extension> backup, just in case. examples: perl -p -e '/admin/' file perl -p -e '^/admin/' file behaves exactly like the basic awk examples. We can also replace stuff : perl -p -e 's/admin/bozo/g' file print all lines matching 'admin', replacing 'admin' with 'bozo'. Now if you just want to replace all occurrences of 'admin' with 'bozo' in the file: perl -pi -e 's/admin/bozo/g' file For instance perl allows you to use a different separator than / for regexps, very useful when manipulating paths: perl -pi -e 's#/some/path/#/different/path/#g' file Of course instead of a simple awk/sed substitute, you can run more elaborate code, and even use perl modules in your one liners with the -M switch: perl -MData::Dumper -n -e 'print Dumper $1 if m/^(admin .w+)/' file Lastly, perl allows you to use q(string) instead of 'string' and qq(string) instead of "string", that avoids lots of escaping when typing in oneliners, so instead of: perl -n -e 'print \'found\' if m/admin/' file Use perl -n -e 'print q(found) if m/admin/' file
101914 12y ago
BEGIN and END are somewhat like what is above the first %% and below the second %% in a yylex "source" file. Or maybe not. I need to review the documentation. This brief AWK intro entices me to try making a similar one for (f)lex using the author's concise format as a model. In any event, Pattern --> Action is common to both programs.
- kazinator 12y ago
  The %% in lex and yacc statically organize the file into different areas. BEGIN and END have run-time semantics: do these things before applying the pattern/actions to the inputs, and do these things afterward.
  - 101914 12y ago
    1. I never mentioned yacc. What relevance does it have to my comment? I typically use (f)lex without yacc/bison to do a similar job as I would use AWK for: text processing. 2. "statically organize the file into different areas" One is a code generator and the other is a scripting language with an interpreter, is that what you mean? In effect, this difference means little to me (except for speed of execution): I store my (f)lex programs as source files that I feed to the (f)lex code generator. Then I compile the generated C code. I store my AWK scripts as source files that I feed to the AWK interpreter. I use both flex and AWK to perform a similar task: text processing. For whatever it is worth, I get better performance from my compiled flex scanners than from my interpreted AWK scripts. But I sometimes use them for the very same text processing jobs. AWK: BEGIN { define variables } pattern-action rules END { stuff to do after EOF } (f)lex: { definitions } user variables %% { rules } pattern-action rules %% { user routines } stuff to do after EOF From the blog: "_BEGIN_, which matches only before any line has been input to the file. This is basically where you can _initiate variables_ and all other kinds of state in your script." From the Lesk and Schmidt: So far only the rules have been described. The user needs additional options, though, to _define variables_ for use in his program and for use by Lex. These can go ... in the _definitions_ section... From the blog: There is also END, which as you may have guessed, will match after the whole input has been handled. This lets you clean up or do some final output before exiting. From Lesk and Schmidt: Another Lex library routine that the user will sometimes want to redefine is yywrap() which is called whenever Lex reaches an end-of-file. I regularly use yywrap in the "user routines" section. It functions much the same way as commands I use in the END section of an AWK script. I guess one can either focus on differences or similarities. I choose the later. I care little about the "intended purpose" of a program. I care more about what a program can actually do.
gesman 12y ago
Excellent tutorial. I'm actually working on (about to finish) free Splunk app that monitors HTTP traffic via Apache logs on WHM/Cpanel based hosting servers and visualizes traffic and activity trends and patterns between IP addresses and sites. Awk would be an excellent tool to quickly play and "debug" logs content alongside with visual tool. Additionally I think I'd want to utilize it for malware detection. + On my "to practice" list.
jacquesm 12y ago
This is written by the same person that wrote 'learn you some erlang for great good'.
Friedduck 12y ago
Were I to have had this instead of the O'Rielly book (Sed & Awk) that I learned from. The time I could have saved. My goal: to be able to write as clearly as this.
lotsofcows 12y ago
Nice write-up! A couple of points: On CentOS, at least, gawk == awk which falsifies "patterns cannot capture specific groups to make them available in the ACTIONS part of the code". Eg: echo "abcdef" | gawk 'match($0, /b(.*)e/, a) { print a[1]; }' You missed the match and not match operators ~ and !~ in your list. Finally, I find people better understand the flexibility of awk when they realise that awk '/bob/ { print }' is shorthand for awk '$0~/bob/ { print $0; }'. This makes it clear that the pattern element is not limited to regexs or matching the whole line.
coliveira 12y ago
The best introduction to awk is its man page. It is such a concise language that you can find pretty much everything you want to learn about it in just a few pages.
- Gracana 12y ago
  The manual page isn't an introduction, it's a reference. It says what it is (the GNU Project's implementation of the AWK programming language), and goes on to talk about how the program is operated and how the language works, but it doesn't explain why or where you'd use this tool That's fine for a reference manual but it's a big omission for an introduction.
  - coliveira 12y ago
    I am not talking about the GNU Info page, but the traditional man page. It is much more concise and useful.
    - tarblog 12y ago
      Can you provide a link to an online version? I'd like to read it
- cbd1984 12y ago
  I actually like the GNU gawk info pages: https://www.gnu.org/software/gawk/manual/ https://www.gnu.org/software/gawk/manual/ Also, TCP/IP with gawk (The gawker's Guide To The Internet?): https://www.gnu.org/software/gawk/manual/gawkinet/ https://www.gnu.org/software/gawk/manual/gawkinet/
known 12y ago
Use gawk for processing large data files
reacweb 12y ago
I use awk only for trivial commands that fit in one line. Here is my last one. awk -F : '/^OPS1:/{print $2}' < machinelist I think the article could add an example of this kind of simple usages.
- aidos 12y ago
  Here's my last one (find half of the free memory to feed into a stress testing tool): stress --vm-bytes $(awk '/MemFree/{printf "%d\n", $2 * 0.5;}' < /proc/meminfo)k --vm-keep -m 1 It's like playing terminal history awk roulette :)
joshbaptiste 12y ago
#awk on irc Freenode great is a great place for general or advanced help
patrickg_zill 12y ago
I have been very happy in using awk, have used it to generate real results. It is fast, a little clunky at first, but since it is a smaller language it is quick to get up to speed on.
proveanegative 12y ago
This seems like a good thread to ask: are there more featureful languages that derive from Awk (edit: i.e., can work in a data driven mode) but don't diverge as much as much as Perl did in terms of syntax?
- kazinator 12y ago
  I made a programming language called TXR which has a "data driven mode", consciously based on the concept, but not derived in any way, and with very different syntax. The data driven mode isn't based on applying patterns to records, but rather patterns to entire text streams. Here is a TXR script I use that transforms the output of the Linux kernel "checkpatch.pl" script into a conventional compiler-like error report that Vim understands: #!/usr/local/bin/txr @(next *stdin*) @(repeat) @type: @message #@code: FILE: @path:@lineno: @(output) @path:@lineno:@type (#@code):@message @(end) @(end) This is "pure TXR": there are no traces of the embedded programming language TXR Lisp. Stuff that isn't @ident or @(syntax) is literal text which is matched. So the block @type: @message #@code: FILE: @path:@lineno: matches a two-line pattern in the checkpatch output, extracting a type, followed by a space and colon, the message, then on the next line a code value prceded by a hash mark, followed by ": FILE: " and then a path delimited by a colon, and a line number terminated by a colon. The following more complicated script scans diff output (usually the output of `git show -p` or `git diff`) and produces an "errors.err" file for Vim such that "vim -q" will navigate over the diffs as a quickfix list, and the messages have some information, like how many lines were added or removed at that point by the diff. Here, there is an overall pattern matching logic for scanning the sections of a diff output: parsing out multiple diffs, and the "hunks" within each diff, with the line number info from the hunk header and such. Some stateful Lisp logic calculates what is needed out of the extracted pieces and produces the output as it goes. #!/usr/local/bin/txr @(next *stdin*) @(do (set *stdout* (open-file "errors.err" "w"))) @(repeat) +++ @nil/@path @ (repeat) @@@@ -@line0,@len0 +@line1,@len1 @@@@@(skip) @ (bind (line start minuses pluses old) (0 nil 0 0 nil)) @ (set line1 @(toint line1)) @ (repeat) @ (cases) -@text @ (do (inc minuses) (set start line) (push text old)) @ (or) @nil @ (do (inc line) (when start (let ((wording (cond ((zerop minuses) `@pluses lines added`) ((zerop pluses) `@minuses lines deleted`) (t `@minuses lines edited to @pluses lines`)))) (put-line `@path:@(+ line1 start):@wording`) (put-lines (nreverse old)) (set minuses 0) (set pluses 0) (set start nil) (set old nil)))) @ (or) +@nil @ (do (if (eql 1 (inc pluses)) (set start line)) (inc line)) @ (or) @ (accept) @ (end) @ (until) @/[^+\- ]/@(skip) @ (end) @ (until) --- @(skip) @ (end) @(end) Example: $ git show -p [ ... ] diff --git a/Makefile b/Makefile index a68fd84..f97b77b 100644 --- a/Makefile +++ b/Makefile @@ -163,14 +163,14 @@ enforce: # Installation macro. # # $1 - chmod perms -# $2 - source file +# $2 - source file(s) # $3 - dest directory # define INSTALL mkdir -p $(3) cp -f $(2) $(3) chmod $(1) $(3)/$(notdir $(2)) - touch -r $(2) $(3)/$(notdir $(2)) + for x in $(2) ; do touch -r $$x $(3)/$$(basename $$x) ; done endef PREINSTALL := : $ git show -p | diff2err.txr $ cat errors.err Makefile:166:1 lines edited to 1 lines # $2 - source file Makefile:173:1 lines edited to 1 lines touch -r $(2) $(3)/$(notdir $(2)) In "vim -q" you are taken to where the changes are, and in the ":cope" window you can see the additional lines that give the original text that was modified, if applicable. For instance, the first item navigates to the line "# $2 - source file(s)" and you know that one line was edited at that point, and the original text was " $2 - source file".
chris_wot 12y ago
You are awesome. That's seriously the easiest bit of technical reading I've ever done. I tip my hat to you!
- pbhjpbhj 12y ago
  Yes, easy reading, as a non-programmer the only thing that lost me was the throw away line in the code comments on "function arguments are call-by-value". I've some recollection that call-by-value and call-by-reference are options here but not entirely clear what the meaning of "call-by-value" is. It probably doesn't matter though - the Wikipedia reference is ever-so slightly to terse for me to understand the implications thoroughly too, http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_value http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_val.... Am I right in saying that if I call a function on $2, say, that unless that variable is explicitly assigned in the function that $2 remains unchanged after the function operates. But that with a call-by-reference the value of the variable itself will be altered? Something along those lines.
  - antsar 12y ago
    Am I right in saying that if I call a function on $2, say, that unless that variable is explicitly assigned in the function that $2 remains unchanged after the function operates. But that with a call-by-reference the value of the variable itself will be altered? You've got it. At least that's what passing a variable by reference vs. by value means in other languages, such as PHP.
- rdtsc 12y ago
  Speaking of awesome, check out other stuff Fred Hebert wrote. Learn You Some Erlang For Great Good: http://learnyousomeerlang.com/ http://learnyousomeerlang.com/ His blog posts are great too: http://ferd.ca/ http://ferd.ca/ And his new (free) book, Erlang In Anger: http://www.erlang-in-anger.com/ http://www.erlang-in-anger.com/
- busterarm 12y ago
  Now do one on sed! :D
baldfat 12y ago
Awk & SED I have a hard time think about one without the other. I think they should be married.
- deleted 12y ago
  [deleted]
- coliveira 12y ago
  I use awk often, but rarely use sed, so I think they don't need to be always together.
- AdmiralAsshat 12y ago
  Awk & Sed are labeled as the following in my mind: - Awk: That thing I use when I need to grep for something over more than one line and/or do some basic transformations. - Sed: That thing with the painful syntax for doing ridiculously complicated regex substitutions. With that said, I find sed much more difficult to use than awk and generally try to avoid it if I can. I'm even prone to just opening the file in vim and executing the replace command through that rather than using sed.
  - seanp2k2 12y ago
    Does noting that you don't have to use / as the separator help with the syntactical pain? I personally always use |, since I'm unlikely to be using that for a pattern. Example: grep foo somefile | sed 's|/path/to/some/file|/new/path|g' Lot nicer than all those picket fences \/\/\/
  - a3n 12y ago
    I would use sed for transformations (simple or complex) where I only care about one transformation (even though you can do multiple): Does the current line match this pattern? Change it to this other thing and move on the next line. I would use awk when there are a handful or more of potential transformations: Does the current line match any of these multiple patterns? Do the action that's defined for each of the patterns, then move on to the next line. If I need to do multiple transformations, and still want to use sed, I find it easiest to create a chain of single sed transformations, piped together. Somewhere in that area a shift to awk (or python) becomes justified.
freditup 12y ago
Very well done article. General question: when would one choose to use an awk script over something more general purpose such as a python or ruby script? To me it would make sense to use the latter in most cases.
- ptaipale 12y ago
  > when would one choose to use an awk script over something more general purpose such as a python or ruby script? When the others are not around. Sometimes you need to work on weird old or stripped-down boxes that don't have python or even perl - but awk is (almost) always there. Like vi.
- kazinator 12y ago
  > when would one choose to use an awk script over something more general purpose such as a python or ruby script? You have almost answered your own question. In situations where you are dealing with munging data which has a structure that is implicitly handled by awk (one or more files, consisting of regularly-delimited records, which break into regularly-delimited fields), it is very difficult to beat awk for succinctness. The second thing is this: Awk has been around for decades and is part of the POSIX standard. A shell script that uses awk commands can be full POSIX compliant and work on any POSIX-like system with minimal changes, without the installation of third-party software. So, with Awk you can do a lot of things that would otherwise require something like Python. In many cases you can do them with clearer code that has less clutter, and your solution is POSIX, to boot. The down side of Awk is that it sacrifices reliability to pander to succinctness. Awk does not detect undefined variables; any variable you mention becomes defined. It has loose arithmetic: this is so you can increment a nonexistent variable or array element by one, and it behaves as if it had been defined with zero. Awk only has local variables in functions, and they are modeled as extra parameters (for which you could pass values, but you don't, unless perpetrating a hack). Basically, whereas you can do actual software engineering in Python and Ruby, you'd be crazy to do it in Awk, even if the lack of libraries and datatypews such isn't an impediment against doing it in Awk.
- tsotha 12y ago
  If you're writing a script that has to run on any Unix or Unix-like system, AWK is a good choice. Because it's going to be there.
OneOneOneOne 12y ago
AWK is very easy for C programmers to learn. You can also try 'info gawk' from the Unix or Cygwin prompt for a good tutorial.
- mseepgood 12y ago
  Or JavaScript programmers. JavaScript's function syntax came from AWK.
  - agumonkey 12y ago
    First time I read that but well the spec says so. http://hepunx.rl.ac.uk/~adye/jsspec11/intro.htm#1006028 http://hepunx.rl.ac.uk/~adye/jsspec11/intro.htm#1006028
- cbd1984 12y ago
  It's also easy for people coming from Perl, given that Perl was designed to be easy for people coming from Unix shell, which includes awk. Python and Ruby aren't that far removed, but the idea of a one-liner as such is alien to the Python mindset, I suppose.
michaelmcmillan 12y ago
Piping things in and out of awk is really powerful and fast! Check out how Gary Bernhardt utilizes different Unix-commands (including awk) to filter out dead links on his blog: http://vimeo.com/11202537 http://vimeo.com/11202537
- greggyb 12y ago
  This is a cool video. I have been using Linux casually for a year and a half now, but haven't progressed too much past the desktop paradigm. I am not uncomfortable in the terminal, but I feel more akin to a foreigner who's mastered a phrasebook, rather than someone able to hold even a rudimentary conversation. Do you (or does anyone else) have more examples similar to this, or resources that would be useful in moving toward this fluency in utilizing command line tools?
  - michaelmcmillan 12y ago
    I would strongly urge you to purchase Gary Bernhardt's screencasts at http://destroyallsoftware.com http://destroyallsoftware.com. It has been extremely influential for me. I've realized that Unix should be looked at as a tool when programming. He also has strong opinions on how to test properly, check this out: https://www.youtube.com/watch?v=tdNnN5yTIeM https://www.youtube.com/watch?v=tdNnN5yTIeM
    - greggyb 12y ago
      These look very interesting. I think I'll probably end up purchasing these.
tieTYT 12y ago
The one thing this is missing is a search/replace example. That's a very common thing to do and annoyingly clunky in Awk.
- shawkinaw 12y ago
  That's why awk is so often used with sed, as awk is great for the first part while sed is very good for the second. For example: echo $BUNCH_OF_TEXT | awk '/amazing/ {print $1, $3}' | sed 's/amazing/mildly interesting/'
- sxv 12y ago
  | awk 'sub(/annoyingly clunky/, "pretty easy")'
- jnazario 12y ago
  a lot of people pipe to sed but you don't need to. you can do regex subs right in awk. see sub and gsub. sub(r, t, s) substitutes t for the first occurrence of the regular expression r in the string s. If s is not given, $0 is used. gsub same as sub except that all occurrences of the regular expression are replaced; sub and gsub return the number of replacements. (this is awk from osx, which i think is nawk)
  - favadi 12y ago
    If you have gawk, use gensub instead.
qodeninja 12y ago
Finally, something useful.
kasperset 12y ago
Awk is also used a lot in Bioinformatics where you need one-liners to extract/format the data. Ofcourse, Perl/Python/R/Ruby can also be used but in some cases Awk is just simple and graceful.
- SilasX 12y ago
  Yes! It shows up a lot among the briefest solutions to Project Rosalind.
vram22 12y ago
Here's a UNIX one-liner I wrote a while ago that uses awk, sed and grep for a real-life need: UNIX one-liner to kill a hanging Firefox process: http://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-hanging-firefox.html http://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-han... The comments on that post are also of interest.
- deleted 12y ago
  [deleted]
kespindler 12y ago
Going to put a shameless plug here for the python awk replacement I made. github.com/kespindler/puffin Instead of learning a new language and new syntax, just use python!
- andre3k1 12y ago
  If only it was as fast as awk. Is it?
  - Russell91 12y ago
    No, the python interpreter itself takes about 0.03 seconds to start up on an i7. Awk and sed and friends are usually about 1/10th of that. So the python interpreter startup time dominates the cost for most simple tasks. It's not enough to notice when you're typing things by hand at the shell, but the difference can be painful if you're writing bash scripts. I use a similar python tool for command line work, but I always go for sed when I'm writing bash scripts that I plan to distribute.
kespindler 12y ago
Going to put a shameless plug here for the python awk replacement I made. github.com/kespindler/puffin Instead of learning a new language and new syntax, just use python!
sea6ear 12y ago
Also Awk has one of the best bite sized tutorial/references out there. Up there with Programming in Lua in terms of awesomeness per page count. http://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp/020107981X/ http://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp...

504 points by craigkerstiens 12y ago

The second comma in the second sentence has got to go. "on files, usually structured" <-- that one. There are some other ones too. Other than that this is really great!

hayksaakian 12y ago

someone should add this to http://learnxinyminutes.com/ http://learnxinyminutes.com/

decisiveness 12y ago

This is probably the most concise introduction to awk I've read. For more extensions[0] and advanced features like arrays of arrays and array sorting, there's also gawk. And for larger files there's performance driven mawk which can drastically increase processing speed[1]. [0]https://www.gnu.org/software/gawk/manual/html_node/Extension-Samples.html https://www.gnu.org/software/gawk/manual/html_node/Extension... [1]http://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ http://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-a...

na85 12y ago

Story time. Back in ~2005 when I was still very new to Linux I had this old Dell that I put Gentoo on. It was a 900MHz Intel Coppermine with a minuscule amount of memory, so things compiled very slow. Being the wise Linux guru that I was back then, I decided that I would emerge (compile) all of X, fluxbox, OpenOffice, and probably 1 or 2 other things. I didn't have distcc set up. It compiled for days. One thing I really did enjoy was watching the cryptic messages crawl by, and one that always mystified me was during the configure stages: checking for gawk... gawk It seemed like a total nonsense word. To this day I still have fond memories every time someone mentions (g)awk.

wazoox 12y ago

I use perl one-liners, thanks to the power of -n, -i and -p switches. perl -n -e <expression> <file> runs the expression on each and every line of <file> (can be STDIN of course). perl -p -e <expression> <file> prints out every line for which the expression returns true. The -i option allows treating files in place. Add -i<extension> to create a <file>.<extension> backup, just in case. examples: perl -p -e '/admin/' file perl -p -e '^/admin/' file behaves exactly like the basic awk examples. We can also replace stuff : perl -p -e 's/admin/bozo/g' file print all lines matching 'admin', replacing 'admin' with 'bozo'. Now if you just want to replace all occurrences of 'admin' with 'bozo' in the file: perl -pi -e 's/admin/bozo/g' file For instance perl allows you to use a different separator than / for regexps, very useful when manipulating paths: perl -pi -e 's#/some/path/#/different/path/#g' file Of course instead of a simple awk/sed substitute, you can run more elaborate code, and even use perl modules in your one liners with the -M switch: perl -MData::Dumper -n -e 'print Dumper $1 if m/^(admin .w+)/' file Lastly, perl allows you to use q(string) instead of 'string' and qq(string) instead of "string", that avoids lots of escaping when typing in oneliners, so instead of: perl -n -e 'print \'found\' if m/admin/' file Use perl -n -e 'print q(found) if m/admin/' file

101914 12y ago

BEGIN and END are somewhat like what is above the first %% and below the second %% in a yylex "source" file. Or maybe not. I need to review the documentation. This brief AWK intro entices me to try making a similar one for (f)lex using the author's concise format as a model. In any event, Pattern --> Action is common to both programs.

kazinator 12y ago
The %% in lex and yacc statically organize the file into different areas. BEGIN and END have run-time semantics: do these things before applying the pattern/actions to the inputs, and do these things afterward.
- 101914 12y ago
  1. I never mentioned yacc. What relevance does it have to my comment? I typically use (f)lex without yacc/bison to do a similar job as I would use AWK for: text processing. 2. "statically organize the file into different areas" One is a code generator and the other is a scripting language with an interpreter, is that what you mean? In effect, this difference means little to me (except for speed of execution): I store my (f)lex programs as source files that I feed to the (f)lex code generator. Then I compile the generated C code. I store my AWK scripts as source files that I feed to the AWK interpreter. I use both flex and AWK to perform a similar task: text processing. For whatever it is worth, I get better performance from my compiled flex scanners than from my interpreted AWK scripts. But I sometimes use them for the very same text processing jobs. AWK: BEGIN { define variables } pattern-action rules END { stuff to do after EOF } (f)lex: { definitions } user variables %% { rules } pattern-action rules %% { user routines } stuff to do after EOF From the blog: "_BEGIN_, which matches only before any line has been input to the file. This is basically where you can _initiate variables_ and all other kinds of state in your script." From the Lesk and Schmidt: So far only the rules have been described. The user needs additional options, though, to _define variables_ for use in his program and for use by Lex. These can go ... in the _definitions_ section... From the blog: There is also END, which as you may have guessed, will match after the whole input has been handled. This lets you clean up or do some final output before exiting. From Lesk and Schmidt: Another Lex library routine that the user will sometimes want to redefine is yywrap() which is called whenever Lex reaches an end-of-file. I regularly use yywrap in the "user routines" section. It functions much the same way as commands I use in the END section of an AWK script. I guess one can either focus on differences or similarities. I choose the later. I care little about the "intended purpose" of a program. I care more about what a program can actually do.

gesman 12y ago

Excellent tutorial. I'm actually working on (about to finish) free Splunk app that monitors HTTP traffic via Apache logs on WHM/Cpanel based hosting servers and visualizes traffic and activity trends and patterns between IP addresses and sites. Awk would be an excellent tool to quickly play and "debug" logs content alongside with visual tool. Additionally I think I'd want to utilize it for malware detection. + On my "to practice" list.

jacquesm 12y ago

This is written by the same person that wrote 'learn you some erlang for great good'.

Friedduck 12y ago

Were I to have had this instead of the O'Rielly book (Sed & Awk) that I learned from. The time I could have saved. My goal: to be able to write as clearly as this.

lotsofcows 12y ago

Nice write-up! A couple of points: On CentOS, at least, gawk == awk which falsifies "patterns cannot capture specific groups to make them available in the ACTIONS part of the code". Eg: echo "abcdef" | gawk 'match($0, /b(.*)e/, a) { print a[1]; }' You missed the match and not match operators ~ and !~ in your list. Finally, I find people better understand the flexibility of awk when they realise that awk '/bob/ { print }' is shorthand for awk '$0~/bob/ { print $0; }'. This makes it clear that the pattern element is not limited to regexs or matching the whole line.

coliveira 12y ago

The best introduction to awk is its man page. It is such a concise language that you can find pretty much everything you want to learn about it in just a few pages.

Gracana 12y ago
The manual page isn't an introduction, it's a reference. It says what it is (the GNU Project's implementation of the AWK programming language), and goes on to talk about how the program is operated and how the language works, but it doesn't explain why or where you'd use this tool That's fine for a reference manual but it's a big omission for an introduction.
- coliveira 12y ago
  I am not talking about the GNU Info page, but the traditional man page. It is much more concise and useful.
  - tarblog 12y ago
    Can you provide a link to an online version? I'd like to read it
cbd1984 12y ago
I actually like the GNU gawk info pages: https://www.gnu.org/software/gawk/manual/ https://www.gnu.org/software/gawk/manual/ Also, TCP/IP with gawk (The gawker's Guide To The Internet?): https://www.gnu.org/software/gawk/manual/gawkinet/ https://www.gnu.org/software/gawk/manual/gawkinet/

known 12y ago

Use gawk for processing large data files

reacweb 12y ago

I use awk only for trivial commands that fit in one line. Here is my last one. awk -F : '/^OPS1:/{print $2}' < machinelist I think the article could add an example of this kind of simple usages.

aidos 12y ago
Here's my last one (find half of the free memory to feed into a stress testing tool): stress --vm-bytes $(awk '/MemFree/{printf "%d\n", $2 * 0.5;}' < /proc/meminfo)k --vm-keep -m 1 It's like playing terminal history awk roulette :)

joshbaptiste 12y ago

#awk on irc Freenode great is a great place for general or advanced help

patrickg_zill 12y ago

I have been very happy in using awk, have used it to generate real results. It is fast, a little clunky at first, but since it is a smaller language it is quick to get up to speed on.

proveanegative 12y ago

This seems like a good thread to ask: are there more featureful languages that derive from Awk (edit: i.e., can work in a data driven mode) but don't diverge as much as much as Perl did in terms of syntax?

kazinator 12y ago
I made a programming language called TXR which has a "data driven mode", consciously based on the concept, but not derived in any way, and with very different syntax. The data driven mode isn't based on applying patterns to records, but rather patterns to entire text streams. Here is a TXR script I use that transforms the output of the Linux kernel "checkpatch.pl" script into a conventional compiler-like error report that Vim understands: #!/usr/local/bin/txr @(next *stdin*) @(repeat) @type: @message #@code: FILE: @path:@lineno: @(output) @path:@lineno:@type (#@code):@message @(end) @(end) This is "pure TXR": there are no traces of the embedded programming language TXR Lisp. Stuff that isn't @ident or @(syntax) is literal text which is matched. So the block @type: @message #@code: FILE: @path:@lineno: matches a two-line pattern in the checkpatch output, extracting a type, followed by a space and colon, the message, then on the next line a code value prceded by a hash mark, followed by ": FILE: " and then a path delimited by a colon, and a line number terminated by a colon. The following more complicated script scans diff output (usually the output of `git show -p` or `git diff`) and produces an "errors.err" file for Vim such that "vim -q" will navigate over the diffs as a quickfix list, and the messages have some information, like how many lines were added or removed at that point by the diff. Here, there is an overall pattern matching logic for scanning the sections of a diff output: parsing out multiple diffs, and the "hunks" within each diff, with the line number info from the hunk header and such. Some stateful Lisp logic calculates what is needed out of the extracted pieces and produces the output as it goes. #!/usr/local/bin/txr @(next *stdin*) @(do (set *stdout* (open-file "errors.err" "w"))) @(repeat) +++ @nil/@path @ (repeat) @@@@ -@line0,@len0 +@line1,@len1 @@@@@(skip) @ (bind (line start minuses pluses old) (0 nil 0 0 nil)) @ (set line1 @(toint line1)) @ (repeat) @ (cases) -@text @ (do (inc minuses) (set start line) (push text old)) @ (or) @nil @ (do (inc line) (when start (let ((wording (cond ((zerop minuses) `@pluses lines added`) ((zerop pluses) `@minuses lines deleted`) (t `@minuses lines edited to @pluses lines`)))) (put-line `@path:@(+ line1 start):@wording`) (put-lines (nreverse old)) (set minuses 0) (set pluses 0) (set start nil) (set old nil)))) @ (or) +@nil @ (do (if (eql 1 (inc pluses)) (set start line)) (inc line)) @ (or) @ (accept) @ (end) @ (until) @/[^+\- ]/@(skip) @ (end) @ (until) --- @(skip) @ (end) @(end) Example: $ git show -p [ ... ] diff --git a/Makefile b/Makefile index a68fd84..f97b77b 100644 --- a/Makefile +++ b/Makefile @@ -163,14 +163,14 @@ enforce: # Installation macro. # # $1 - chmod perms -# $2 - source file +# $2 - source file(s) # $3 - dest directory # define INSTALL mkdir -p $(3) cp -f $(2) $(3) chmod $(1) $(3)/$(notdir $(2)) - touch -r $(2) $(3)/$(notdir $(2)) + for x in $(2) ; do touch -r $$x $(3)/$$(basename $$x) ; done endef PREINSTALL := : $ git show -p | diff2err.txr $ cat errors.err Makefile:166:1 lines edited to 1 lines # $2 - source file Makefile:173:1 lines edited to 1 lines touch -r $(2) $(3)/$(notdir $(2)) In "vim -q" you are taken to where the changes are, and in the ":cope" window you can see the additional lines that give the original text that was modified, if applicable. For instance, the first item navigates to the line "# $2 - source file(s)" and you know that one line was edited at that point, and the original text was " $2 - source file".

chris_wot 12y ago

You are awesome. That's seriously the easiest bit of technical reading I've ever done. I tip my hat to you!

pbhjpbhj 12y ago
Yes, easy reading, as a non-programmer the only thing that lost me was the throw away line in the code comments on "function arguments are call-by-value". I've some recollection that call-by-value and call-by-reference are options here but not entirely clear what the meaning of "call-by-value" is. It probably doesn't matter though - the Wikipedia reference is ever-so slightly to terse for me to understand the implications thoroughly too, http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_value http://en.wikipedia.org/wiki/Evaluation_strategy#Call_by_val.... Am I right in saying that if I call a function on $2, say, that unless that variable is explicitly assigned in the function that $2 remains unchanged after the function operates. But that with a call-by-reference the value of the variable itself will be altered? Something along those lines.
- antsar 12y ago
  Am I right in saying that if I call a function on $2, say, that unless that variable is explicitly assigned in the function that $2 remains unchanged after the function operates. But that with a call-by-reference the value of the variable itself will be altered? You've got it. At least that's what passing a variable by reference vs. by value means in other languages, such as PHP.
rdtsc 12y ago
Speaking of awesome, check out other stuff Fred Hebert wrote. Learn You Some Erlang For Great Good: http://learnyousomeerlang.com/ http://learnyousomeerlang.com/ His blog posts are great too: http://ferd.ca/ http://ferd.ca/ And his new (free) book, Erlang In Anger: http://www.erlang-in-anger.com/ http://www.erlang-in-anger.com/
busterarm 12y ago
Now do one on sed! :D

baldfat 12y ago

Awk & SED I have a hard time think about one without the other. I think they should be married.

deleted 12y ago
[deleted]
coliveira 12y ago
I use awk often, but rarely use sed, so I think they don't need to be always together.
AdmiralAsshat 12y ago
Awk & Sed are labeled as the following in my mind: - Awk: That thing I use when I need to grep for something over more than one line and/or do some basic transformations. - Sed: That thing with the painful syntax for doing ridiculously complicated regex substitutions. With that said, I find sed much more difficult to use than awk and generally try to avoid it if I can. I'm even prone to just opening the file in vim and executing the replace command through that rather than using sed.
- seanp2k2 12y ago
  Does noting that you don't have to use / as the separator help with the syntactical pain? I personally always use |, since I'm unlikely to be using that for a pattern. Example: grep foo somefile | sed 's|/path/to/some/file|/new/path|g' Lot nicer than all those picket fences \/\/\/
- a3n 12y ago
  I would use sed for transformations (simple or complex) where I only care about one transformation (even though you can do multiple): Does the current line match this pattern? Change it to this other thing and move on the next line. I would use awk when there are a handful or more of potential transformations: Does the current line match any of these multiple patterns? Do the action that's defined for each of the patterns, then move on to the next line. If I need to do multiple transformations, and still want to use sed, I find it easiest to create a chain of single sed transformations, piped together. Somewhere in that area a shift to awk (or python) becomes justified.

freditup 12y ago

Very well done article. General question: when would one choose to use an awk script over something more general purpose such as a python or ruby script? To me it would make sense to use the latter in most cases.

ptaipale 12y ago
> when would one choose to use an awk script over something more general purpose such as a python or ruby script? When the others are not around. Sometimes you need to work on weird old or stripped-down boxes that don't have python or even perl - but awk is (almost) always there. Like vi.
kazinator 12y ago
> when would one choose to use an awk script over something more general purpose such as a python or ruby script? You have almost answered your own question. In situations where you are dealing with munging data which has a structure that is implicitly handled by awk (one or more files, consisting of regularly-delimited records, which break into regularly-delimited fields), it is very difficult to beat awk for succinctness. The second thing is this: Awk has been around for decades and is part of the POSIX standard. A shell script that uses awk commands can be full POSIX compliant and work on any POSIX-like system with minimal changes, without the installation of third-party software. So, with Awk you can do a lot of things that would otherwise require something like Python. In many cases you can do them with clearer code that has less clutter, and your solution is POSIX, to boot. The down side of Awk is that it sacrifices reliability to pander to succinctness. Awk does not detect undefined variables; any variable you mention becomes defined. It has loose arithmetic: this is so you can increment a nonexistent variable or array element by one, and it behaves as if it had been defined with zero. Awk only has local variables in functions, and they are modeled as extra parameters (for which you could pass values, but you don't, unless perpetrating a hack). Basically, whereas you can do actual software engineering in Python and Ruby, you'd be crazy to do it in Awk, even if the lack of libraries and datatypews such isn't an impediment against doing it in Awk.
tsotha 12y ago
If you're writing a script that has to run on any Unix or Unix-like system, AWK is a good choice. Because it's going to be there.

OneOneOneOne 12y ago

AWK is very easy for C programmers to learn. You can also try 'info gawk' from the Unix or Cygwin prompt for a good tutorial.

mseepgood 12y ago
Or JavaScript programmers. JavaScript's function syntax came from AWK.
- agumonkey 12y ago
  First time I read that but well the spec says so. http://hepunx.rl.ac.uk/~adye/jsspec11/intro.htm#1006028 http://hepunx.rl.ac.uk/~adye/jsspec11/intro.htm#1006028
cbd1984 12y ago
It's also easy for people coming from Perl, given that Perl was designed to be easy for people coming from Unix shell, which includes awk. Python and Ruby aren't that far removed, but the idea of a one-liner as such is alien to the Python mindset, I suppose.

michaelmcmillan 12y ago

Piping things in and out of awk is really powerful and fast! Check out how Gary Bernhardt utilizes different Unix-commands (including awk) to filter out dead links on his blog: http://vimeo.com/11202537 http://vimeo.com/11202537

greggyb 12y ago
This is a cool video. I have been using Linux casually for a year and a half now, but haven't progressed too much past the desktop paradigm. I am not uncomfortable in the terminal, but I feel more akin to a foreigner who's mastered a phrasebook, rather than someone able to hold even a rudimentary conversation. Do you (or does anyone else) have more examples similar to this, or resources that would be useful in moving toward this fluency in utilizing command line tools?
- michaelmcmillan 12y ago
  I would strongly urge you to purchase Gary Bernhardt's screencasts at http://destroyallsoftware.com http://destroyallsoftware.com. It has been extremely influential for me. I've realized that Unix should be looked at as a tool when programming. He also has strong opinions on how to test properly, check this out: https://www.youtube.com/watch?v=tdNnN5yTIeM https://www.youtube.com/watch?v=tdNnN5yTIeM
  - greggyb 12y ago
    These look very interesting. I think I'll probably end up purchasing these.

tieTYT 12y ago

The one thing this is missing is a search/replace example. That's a very common thing to do and annoyingly clunky in Awk.

shawkinaw 12y ago
That's why awk is so often used with sed, as awk is great for the first part while sed is very good for the second. For example: echo $BUNCH_OF_TEXT | awk '/amazing/ {print $1, $3}' | sed 's/amazing/mildly interesting/'
sxv 12y ago
| awk 'sub(/annoyingly clunky/, "pretty easy")'
jnazario 12y ago
a lot of people pipe to sed but you don't need to. you can do regex subs right in awk. see sub and gsub. sub(r, t, s) substitutes t for the first occurrence of the regular expression r in the string s. If s is not given, $0 is used. gsub same as sub except that all occurrences of the regular expression are replaced; sub and gsub return the number of replacements. (this is awk from osx, which i think is nawk)
- favadi 12y ago
  If you have gawk, use gensub instead.

qodeninja 12y ago

Finally, something useful.

kasperset 12y ago

Awk is also used a lot in Bioinformatics where you need one-liners to extract/format the data. Ofcourse, Perl/Python/R/Ruby can also be used but in some cases Awk is just simple and graceful.

SilasX 12y ago
Yes! It shows up a lot among the briefest solutions to Project Rosalind.

vram22 12y ago

Here's a UNIX one-liner I wrote a while ago that uses awk, sed and grep for a real-life need: UNIX one-liner to kill a hanging Firefox process: http://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-hanging-firefox.html http://jugad2.blogspot.in/2008/09/unix-one-liner-to-kill-han... The comments on that post are also of interest.

deleted 12y ago
[deleted]

kespindler 12y ago

Going to put a shameless plug here for the python awk replacement I made. github.com/kespindler/puffin Instead of learning a new language and new syntax, just use python!

andre3k1 12y ago
If only it was as fast as awk. Is it?
- Russell91 12y ago
  No, the python interpreter itself takes about 0.03 seconds to start up on an i7. Awk and sed and friends are usually about 1/10th of that. So the python interpreter startup time dominates the cost for most simple tasks. It's not enough to notice when you're typing things by hand at the shell, but the difference can be painful if you're writing bash scripts. I use a similar python tool for command line work, but I always go for sed when I'm writing bash scripts that I plan to distribute.

kespindler 12y ago

Going to put a shameless plug here for the python awk replacement I made. github.com/kespindler/puffin Instead of learning a new language and new syntax, just use python!

sea6ear 12y ago

Also Awk has one of the best bite sized tutorial/references out there. Up there with Programming in Lua in terms of awesomeness per page count. http://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp/020107981X/ http://www.amazon.com/AWK-Programming-Language-Alfred-Aho/dp...