11 ms·
Why you should learn at least a little bit of Awk
- d4rt 16y agoI can't recommend the Sed & Awk book enough. Regular expressions are my favourite secret weapon; So many problems are made simple by regular expressions and so few people (outside of IT) know of them.
- telemachos 16y agoAfter I recommended the Sed & Awk book here a few months ago, silentbicycle countered[1] that The Awk Programming Language[2] (by Aho, Kernighan and Weinberger) was much better. I was curious enough that I bought and read it just at the end of summer. It really is excellent. Highly, highly recommended. [1] http://news.ycombinator.com/item?id=1403376 http://news.ycombinator.com/item?id=1403376 [2] http://cm.bell-labs.com/cm/cs/awkbook/ http://cm.bell-labs.com/cm/cs/awkbook/
- deleted 16y ago[deleted]
- signa11 16y agofrom what i remember, "the unix programming environment" by pike et al is also pretty good, and contains a basic introduction to most of the unix utilities.
- silentbicycle 16y agoYou can't go wrong with any programming books Brian Kernighan co-wrote, really. I have _The ANSI C Programming Language_ ("K&R"), _The Practice of Programming_, _The AWK Programming Language_, and _The Unix Programming Environment_, and they're all great. Concise, with a lot of depth that reveals itself on repeat reading. Ieursalimschy's _Programming in Lua_ ("PiL") was written in a similar style. I recommend it quite highly, too. Great language, great programming book. Also, the PSD, SMM, and USD books (_4.4BSD Programmer's Supplementary Documents_, etc.) are dry, but also have excellent introductions to several classic Unix tools. They're included as documentation in some BSD installations, and should be easy to find otherwise. The intros to lex and yacc are particularly good.
- telemachos 16y agoI'm glad you saw this thread. It's always nice to find out somebody actually paid attention to (and appreciated) some advice you put out on the interwebs.
- silentbicycle 16y ago:) Awk threads always seem to get my attention.
- grease 16y agoI went through the article and tried the stuff on the log files on my web-server. Useful stuff.
- Kliment 16y agoAwk is a great and oft-forgotten tool. Not only is it useful, the awk way of thinking about stream processing generalizes nicely to a bunch of other areas. You have a block that runs before anything else happens, a block run just before the program exits, and a block run for every piece of input. In awk, the input is a line of text, but nothing stops you from generalizing this to say a frame from a video (split into channels in various colorspaces, fed through a processing pipeline, returning another, processed image), a sound frame, a sensor measurement...
- derefr 16y agoGeneral term for this: "Hylomorphism", defined as the composition of an anamorphism (a generator function) and a catamorphism (a fold/map-reduce function.) The initial base case of the generator runs BEGIN{}, and the terminal base case of the fold runs END{}.
- silentbicycle 16y agoActually, I think the general term for this is a "pipe". (I've seen it called "generate and test [programming]" in Prolog books, but that's specific to a filtering pipe.)
- jakevoytko 16y ago"nothing stops you from generalizing this to say a frame from a video" This is 100% true. A coworker of mine implemented an elevation-bitmap-to-3d-model conversion tool in 160 lines of Awk. It ran faster than our "good" Matlab tool by a factor of 10. Awk (or Perl) doubles the usefulness of Unix. Most of the common commands in Unix are query commands. When you need to start manipulating queried data, Awk is where the rubber meets the road. Piping data through the shell stops being read-only, and becomes interactive.
- silentbicycle 16y ago> It ran faster than our "good" Matlab tool by a factor of 10. Could you give a bit more details there? I don't have any experience with matlab, but I tend to think of awk as fast to write code in (and start up), though not particularly fast in execution. (Roughly on par with Python, i.e., usually good enough.)
- stevefink 16y agoThat and most places with a clue that are hiring competent sys admins will expect at least some knowledge of sed and awk.
- redcap 16y agoIt's been a while since I've touched awk, but I've certainly got a lot of use from it when ripping data out of logfiles and using it elsewhere on the command-line. I can recommend this text file of awk one-liners: http://www.pement.org/awk/awk1line.txt http://www.pement.org/awk/awk1line.txt And for completeness, here's one for sed: http://sed.sourceforge.net/sed1line.txt http://sed.sourceforge.net/sed1line.txt
- silentbicycle 16y agoMany people consider Perl to be the next evolution of awk, but I prefer to think of awk as (just) the essentials of Perl. Perl has CPAN, etc., but for quick string hackery, everything you need fits in one tiny awk reference. Its design hasn't sprawled the way Perl's has. (Except for gawk. The FSF does bloat better than anyone.) It's incredibly handy, yet the language is small enough that you can learn most of it in an evening, with just a bit longer if you don't know regular expressions.
- _delirium 16y agoI definitely prefer awk versus perl for one-liners, with some sed thrown in. Perl does have some command-line switches to ease certain kinds of one-liners, but it just feels more verbose for that kind of interactive use (feels more oriented towards writing scripts). I do tend to use Perl for things where speed matters, though, especially with large amounts of data going through a regex--- Perl's regex engine seems considerably faster than any awk (or especially sed) I've tested, at least on a few examples I've ported in the past. I was surprised once to get an 8x speedup by porting a 3-line sed script to a 3-line perl script (it was basically doing s/ABC/A\nC/g on a multigigabyte file). I've heard mawk can be speed-competitive with Perl, though.
- silentbicycle 16y agoSame here, but I prefer Lua to Perl, and Lua's LPEG (http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html) compares very favorably to common regexp implementations. (There are benchmarks in the paper.) It's based on PEGs, a different formalism than regular expressions. PEGs are more expressive - they're able to handle balanced, recursive structures, for example. LPEG is a nice middle ground between regular expressions and a full LALR(1) parser.
- swift 16y agoI'm not sure if "middle ground" is quite right; PEGs and CFGs can express a different set of languages, and each has their own advantages. Probably the most important tradeoff is that with PEGs you gain infinite lookahead and negation, but you lose left recursion and the ability to express ambiguity.
- SpaceHobo 16y agoI had great fun writing the traditional "Cloak of Darkness" exercise for Interactive Fiction in pure AWK: http://zork.net/~nick/loyhargil/if/if.awk http://zork.net/~nick/loyhargil/if/if.awk For comparison, here are all the published examples of this exercise in a variety of systems: http://www.firthworks.com/roger/cloak/ http://www.firthworks.com/roger/cloak/ I won't say it's the best tool for this job, but I feel that the awkishness provides a certain elegance to some aspects.
- cbernini 16y agoSince I was introduced to AWK I didn't look back, 80% I have to do on the command line end up using AWK for it.
- awakeasleep 16y agoNote that the first code he writes on the page awk "{print $0}" does not work. Awk programs need single quotes to prevent bash expansion.
- beza1e1 16y agoDepends on the shell one uses, i think. On the other hand, who doesn't use bash these days.
- silentbicycle 16y agoSee: http://news.ycombinator.com/item?id=1669409 http://news.ycombinator.com/item?id=1669409 (Pet peeve of mine.)
- b3n 16y agoAssuming you're using the Bash shell.
- hernan7 16y agoSame thing happens on Bourne shell, or Korn shell.
- gregable 16y agoDoh. Fixed.
- ralph 16y agoThis can be simplified. :-) awk 1
- chaostheory 16y agoI can understand the usefulness of awk if say only C, C++, Java/C# existed, but given that it's just easy and fast as Awk to code something useful and powerful using something modern like either Ruby and Python; I just fail to see the point. O yeah and let's not forget Perl.
- loup-vaillant 16y agoAwk is so small that you can be productive in half an hour. It's so concise that most useful programs are easy little one-liners. It's so fast that you can trust it with massive data crunching. In other words, awk is unbeatable for stream crunching. (That's the point of being domain specific, by the way.)
- chaostheory 16y ago"Awk is so small that you can be productive in half an hour. It's so concise that most useful programs are easy little one-liners." I can say the same for ruby and python (and perl). From personal experience, as an awk script/program becomes more important - it will evolve with more requirements and it will start to be clunky. It just isn't practical to stick with it since you'll eventually need the features/libraries that the other languages have. Given the choices we have today, why even start with awk? On the performance side, you can always just use Lua if that's really important.
- silentbicycle 16y agoThe major benefit with awk is that it runs as a pattern recognizing/processing filter by default, so it handles certain common problems in very little code, and fits particularly well in Unix shell pipelines. I'm also a big fan of structuring code in terms of pattern-matching. (I wrote an Erlang-style pattern matching library for Lua, btw: http://github.com/silentbicycle/tamale/ http://github.com/silentbicycle/tamale/ ) I write a lot of little awk scripts, but if they grow past ~5 lines, they usually get rewritten in Lua. (Perhaps eventually with inner loops in C.) Still, Awk is simple and useful enough that it's still worth knowing.
- jdp23 16y agoBack in the 80s I wrote a 500-line program analysis tool in Awk. One day the woman I was going out with handed me a printout I had left at her place, saying something along the lines of "here's your awk code". She wasn't a programmer so I was stunned that she knew it was Awk, and very impressed too. Years later I ran into Brian Kernighan at a conference and told him the story, ending it with "and that's when I knew she was the woman for me." He looked at me like I was nuts.
- SkyMarshal 16y agoGreat story, but don't leave us hanging. Just how did she know it was awk code?
- jdp23 16y agoone of the women she worked with used awk a lot for munging data and simple reports from their pre-SQL database ... it's pretty recognizable :-)
- poome 16y agoIs she in the kitchen right now?
- brendano 16y agoAwk can also be faster than (naively written) C++. http://anyall.org/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ http://anyall.org/blog/2009/09/dont-mawk-awk-the-fastest-and...
- silentbicycle 16y agoThat's not news, though. Better algorithms trump constant factors, and what is "naively written C++" if not a murder of bad algorithm choices? I'd bet that people get shit done 10x+ times faster in awk/lua/python/ruby/lisp/whatever until having to work with nasty C++-specific libraries dominates, though. (C is friendlier that way.)
- nwmcsween 16y agoI know this goes against what is said here but I hate awk. The syntax is so convoluted that it seems parts were picked with whatever was reasonable at the time. It's like I'm banging on rocks in a cave somewhere every time I have to work with bash, awk and related tools. In fact I did a quick bash script of pattern matching some files, moving them, resizing and compressing then uploading and it took three days to read man pages, parsing, etc till I got fed up, used ruby and had it done in under an hour.