19 ms·
The Awk Programming Language (1988) [pdf]
- asicsp 8y agoSee also: http://www.faqs.org/faqs/computer-lang/awk/faq/ http://www.faqs.org/faqs/computer-lang/awk/faq/ from 2002 For latest manual/book: https://www.gnu.org/software/gawk/manual/ https://www.gnu.org/software/gawk/manual/
- coliveira 8y agoI consider awk to be the most useful and underused language in the UNIX ecosystem. I use it daily to analyze, transform, and assemble data, and it always blows my mind that so few people really know how to use it at a decent level. This is an excellent book to give a real idea of what awk is capable of.
- taeric 8y agoI think my hiccup is that it seems like good awk programmers are expected to be able to pass the full bloody program in as a simple string argument in a bash one-liner. Which is clearly not tenable. There are a few things I have done which felt more difficult than they should. However, the vast majority of dealing with structured data using awk really has been a pleasant surprise.
- asicsp 8y agoone-liners are fun though, especially if you like code golfing.. if you're interested, you could give my tutorial(https://github.com/learnbyexample/Command-line-text-processing/blob/master/gnu_awk.md https://github.com/learnbyexample/Command-line-text-processi...) a try it has over 300 examples..
- ageofwant 8y agoThanks for that mate. I'm a regular customer your work is greatly appreciated.
- asicsp 8y agothanks for the feedback :) happy learning
- natecavanaugh 8y agoI gotta say, I really enjoyed skimming through this, and saving it for future reference. I can already tell there are some basic things that I should've memorized with all the one liners I've used, but because most were cobbled together for the immediate task (and many online forum searches assume a lot of prior knowledge), so t this presentation, and cookbook style references really help me hold the practical benefits in memory alot longer, so thanks again!
- asicsp 8y agothanks, that means a lot to me :) if you are interested in text processing in general, do check the other chapters for grep/sed/perl/sort/paste/pr/etc I learned a lot writing them up and marvel at the speed and functionality provided by these tools.. for many cases, it is lot easier to write a bash script combining these tools than to write a Perl/Python/Ruby script..
- xpil 8y agoAhh, what a useful link! Bookmarking immediately. I use AWK like once a year but when I do, I usually struggle a lot. Your examples are really helpful!
- asicsp 8y agothanks, it's a reference for me as well - not easy to remember all the features and tricks
- schoen 8y agoI think some of the examples in popular documentation use awk -f to read a program from a file, so I don't feel like that's regarded as an unreasonable way to run awk code. It's true that many programs might well just have a single action (or a single action plus a BEGIN and END), but I'm not sure that resorting to awk -f will give great offense.
- coliveira 8y agoAwk makes it easy to write one liners, but it is also very easy to save your script to a file and run it with the -f option.
- Stratoscope 8y agoAwk did have this reputation of hard-to-read one liners, but it was also always possible to write readable code in it much as you would do in any language. My favorite bit of Awk code that I ever wrote was this print formatting program for the HP LaserJet II: https://github.com/geary/awk/blob/master/LJPII.AWK https://github.com/geary/awk/blob/master/LJPII.AWK It printed source code in a "2-up" format, with two pages printed side by side in landscape mode. It looked for the kind of separators that I and many people were fond of back in those days (the source code is a good example) and converted those to graphic boxes. And it tried to be smart about things like breaking pages before a comment box, not after. I wish I still have a sample printout handy. For 300 lines of what I thought was fairly cleanly written Awk code, it made some nicely formatted printouts.
- oblio 8y agoThat's very clean looking code and I think it would be considered so regardless of the language it was written in, IMHO. Nice! :)
- pmarin 8y agowhat I usually do is to wrap the awk script in to a sh function. I never do one-liners unless it is a well known idiom.
- ajross 8y ago> it always blows my mind that so few people really know how to use it at a decent level Not nearly as surprising as it is to me that now that most developers have forgotten perl, they're turning to awk as an inspiring example of a bygone era. Seriously: perl basically replaced awk in the mid-90's. It absorbed all the great lessons and added two or three dozen innovations. But it had scary syntax, so everyone used python (which did not replace awk very effectively) and forgot perl. So now we're back at awk. And the scary syntax is all in the Rust world.
- kqr 8y agoWhat I like about AWK is that it is described by a few pages of the POSIX standard. If Perl was that ubiquitous and simple, I would prefer it over AWK.
- ajross 8y agoTo be clear: I like that about it too. But in the real world, we want tooling that does more stuff. In the mid-90's, "everyone" knew perl, because it was the best choice for problems in this space. And in a world where everyone knows perl, there is no place for awk. But now we live in a world where no one knows perl, and languages like python and javascript are clumsy and weird in this space. And that makes awk look clever and elegant. All I'm saying is that in perl-world (which was a real place!), awk wasn't clever and elegant, it was stale and primitive. And that to me is more surprising than awk's cleverness.
- fiddlerwoaroof 8y agoI think people underestimate the importance of medium-powered tools. Perl is great, but awks limitations make it easier to write in a way that the next person can maintain.
- fiddlerwoaroof 8y agoThat being said, the couple times I’ve paired with a perl wizard were revelatory.
- deleted 8y ago[deleted]
- busterarm 8y agoI think my tendency to reach for grep/sed/awk might be the thing that led me away from software development full-time and into the ops world. Too many times I encountered 200+ line Python scripts that I could replace with an Awk one-liner and a cron job. My lazy habit in my Ruby work eventually became to parse JSON/XML/whatever into usable text, shelling out to sed/awk and working with the results. This would not only save me LoC, but was less error-prone.
- boobsbr 8y agoI never got past using awk for string replacement that I couldn't pull off as a regex in grep...
- deleted 8y ago[deleted]
- Annatar 8y agoI use AWK daily; it’s my workhorse programming language and automation tool. Small, fast, no dependencies and portable. Extremely powerful programming language for big data processing and data driven programming, I often use it to generate shell scripts based on some arbitrary data. With no memory management and providng hash arrays, AWK is an absolute delight to program in. Using it with the functional programming paradigm makes it even more powerful. It blows my mind that Aho, Weinberger and Kernighan managed to design something so versatile and yet so small and fast. I’ve also been using it to replace Python scripts with a ratio of Python:AWK being anywhere from 3:1 to 10:1 as far as lines of code needed to get the same tasks done.
- gnuvince 8y agoAgreed. I learned awk a bit late in my Linux journey (started using Linux in 1999, learned awk in 2016), but I can't imagine not using it daily now. Great tool to manipulate data, and it's great for when you want a Unix tool that doesn't exist (e.g., computing the sum or mean of a stream of numbers).
- mdip 8y agoit always blows my mind that so few people really know how to use it at a decent level I'll second this -- it's by far the most used swiss-army knife in my arsenal[0]. A little part of me dies when I see scripts doing grep|awk, using awk simply to print a column. After taking some time to actually read the gawk user manual a few years ago, I've found that I can do most things in awk that I was previously using grep/sed (and to a lesser extent) perl/python for in the shell. In my environment, it also comes with the benefit that I know the awk code I'm writing is supported by the version of gawk that's installed on every server/computer I come into contact with -- no having to ensure the right python/perl is available. One thing I wish there was better documentation on, though, is creating complete gawk scripts. With @include directives coupled with AWK_PATH, it's really convenient to build up a handful of utility functions that do frequent things and I've found, on several occasions, that I end up writing an .awk script instead of a bash/zsh script and it ends up being a much more straight-forward set of code. [0] Well, specifically, the 'gawk' variant
- ausjke 8y agowhy not write something about the @include directives like a blog or some posts, that will be awesome.
- wernsey 8y agoOne thing I also really like about Awk is that you can keep 99% of the language in your head. With Awk it is seldom necessary to even reference the man page. I don't have to interrupt myself with looking up the syntax or the order of the parameters to a function.
- gitgacutils 8y agoawk is a fundamental part of the unix ecosystem. It's in the hall of fame of utils along with the likes of grep. If you use unix or linux in any semi-serious capacity, you are familiar with awk.
- znpy 8y agoI printed this book and went through it and imho just skimming through all of it is worth it: just understanding how it works beyond the basic '{print $2}' is immensely worth it, and being exposed to some 'advanced' techniques gives you a set of techniques that you can reuse in your daily chores (in particular if you're a sysadmin).
- deleted 8y ago[deleted]
- totalperspectiv 8y agoThis book is worth the read. Just to get in the mindset of the authors. I wish that more programming books could be as concise and useful at the same time.
- olskool 8y agoOver the years I've written too many awk one liners to count. Most of them look ugly - hell awk makes Perl look elegant - but having awk in your toolkit means that you don't have to drop out of the shell to extract some weird shit out of a text stream. Thanks Aho Weinberger and Kernigan!
- deleted 8y ago[deleted]
- jph 8y agoAwk is great for quick command line scripts and also for running on a very wide range of systems. I recently wrote a simple statistics tool using Awk to calculate median, variance, deviation, etc. and people say the code is readable and good for seeing the simplicity of Awk. https://github.com/numcommand/num/blob/master/bin/num https://github.com/numcommand/num/blob/master/bin/num
- hawski 8y agoAnd I'm still waiting for the structural regular expressions version of awk [0]. I very much like awk, I prefer it over sed, because it's easy to read. Also proper man page is all one needs. But I find myself many times doing something like this: match($0, /regex/) { x = substr($0, RSTART, RLENGTH) if(match(x, /regex2/)) { ... } else if(match(x, /regex3/)) { ... Then I sometimes want to mix and match those strings. Or do some math on a matched number. It's a bit tedious in awk. [0] http://doc.cat-v.org/bell_labs/structural_regexps/ http://doc.cat-v.org/bell_labs/structural_regexps/
- orionblastar 8y agoIn 1996 I worked as a federal contractor for a US Army base. They had different Unix systems locked down for security reasons. Had Awk and Sed to work with and ordered the books from Amazon. Oracle databases and other databases exported data in fixed width files and I had to download from several Nix systems to import into one general Nix system using Oracle and then a DOS based Clipper 5 system and an Access 2.0 Windows system and they all had to get the same results. If not for Awk I could not filter the files from the Nix systems.
- thomastjeffery 8y agoSaying "Nix" instead of "*nix" is confusing because of the Nix package manager[0] [0]https://nixos.org/ https://nixos.org/
- oblio 8y agoI recently had a sort of "contest" with someone for parsing the output of a tool. I had to parse some text output into a tree structure. The other person wrote it in awk, quite quickly. After writing my own version in Python (my version was waaaay over-engineered), I decided to blatantly rip-off the awk solution and re-implement it in Python. It was almost as simple and as short. Awk is much more compact as a language, but also way more limited. And it still has its quirks and a certain volume of information you have to gather. I'd say it's more worthwhile to learn Python instead, because you'll be able to use it for other purposes.
- rurban 8y agoIf you want a fast awk, use mawk. https://github.com/mikebrennan000/mawk-2 https://github.com/mikebrennan000/mawk-2
- Annatar 8y ago...Or transpile it to ANSI C with AWKA.
- davidgould 8y agoYes. mawk is shockingly fast. In my perfect world mawk would have some of the gawk extensions, and it would have a csv reader mode to properly split csv into $1...$NF. Because that would be the killer tool.
- rurban 8y agoThe new mawk has it, just the old Debian version not yet.
- davidgould 8y agoWhere is the new mawk please?
- florinutz 8y agobold was called heavy, lol
- hi41 8y agoI find awk so beautiful. I written many scripts in awk. It is so good at data transformation. I used it write a script to delete old and unused records from tables. The book is so beautifully written with amazing clarity of thought.
- carlmr 8y agoI just skimmed it in 30 minutes. I feel I can write some simple stuff now. Except for all the examples it doesn't feel that overwhelming.
- anoonmoose 8y agoI wanted to know what the language looked like so I went to the first example in the book and found this: This is the kind of job that awk is meant for, so it's easy. Just type this command line: awk '$3 > 0 { print $1, $2 * $3 }' emp.data
- ianmcgowan 8y agoI had this exact book and used awk on DOS/Novell in the early 90's when scripting choices were pretty scarce. The writing is tremendous - a model of clarity, and worth reading just for that. Anything with Kernighan, Pike or Plauger as author is worth checking out just for the example of clear thinking.
- segmondy 8y agoI've this book and highly recommend. I have referenced it numerous times to pull out text manipulation wizardly that stunned others.
- technofiend 8y agoLol. This is the IT equivalent of the hairy dog story about the itemized million dollar invoice: $1 - drill a hole, $999,999 - knowing where to drill the hole. And by that I mean sometimes it seems easy to solve a problem because you have the skill to do so. It looks easy but that's only because of the time invested in making it easy for you. For anyone else the challenge remains.
- rbc 8y agoI have this book. I use awk daily to do analysis of Suricata logs. It's great for querying structured text.
- lasermike026 8y agoAwk has it's uses. If you use the command line you'll probably use Awk occasionally. I don't get the Perl hate. Perl's unpopularity may have something to do with some of the languages design choices. I think what really killed it was Perl coders. Some of the worst code I've seen happened to be written in Perl. If you follow clean code principles Perl is fine. Mojolicious is an awesome framework. I like it a lot. Today I code Python and C. I used to code Ruby and before that Perl. I loved Ruby's syntax but Ruby seems to be waning. I'm looking forward to coding in Go. I'll be coding Javascript but I'm not looking forward to it. Use the tool that fits the job. I have no loyalties to any programming language.
- mxschumacher 8y agothe most recent version(s) of Javascript are quite pleasurable. A lot has happened in the last 4 years.
- cphoover 8y agoDon't judge JavaScript before you try it... it has a lot of good tooling around it, and a enthusiastic, and large community. There is a reason companies like Netflix, and Walmart are heavily invested in using Node.js for their internal services.
- lasermike026 8y agoYes, I'm getting there. I'm going to embrace it.
- Zenst 8y agoIt's like shells or editors. Many `older` beards are mindful of the aspect that not all tools are equal in that most systems had some tools shipped as default and others you had to go and install yourself. Thus they became wise though experience of sticking to using tools that did the job and would be more commonly available. Hence they all grew up with the likes of vi,sh and awk. They did the job and worked well. Yes alternatives became more acesable and now most systems ship with all the flavour of shell and editor as they class the likes of bash,ksh,csh... and emac/perl to be part of those essential common installs. This and space/storage less of a factor as it was decades ago. Really gets down to personal taste now more than it did in the past, but if a tool works, does the job and the alternatives don't offer any gains, then they stick with that. Not saying the alternatives are bad or worse in any way, or indeed better, they are just different and in some cases, maybe better. At least the core reason of them always being there instead of having to add another dependency and risk factor to a system have become moot these days. TL;DR some older tools more guaranteed to be upon all systems as a lowest common denominator in the older days than now and legacy always outlives the machine, hence we still run COBOL today as it works, maybe better solutions but rebuilding rome overnight still avoided.
- mxschumacher 8y agoI love the typography
- kps 8y agoThat's troff at work.
- tux1968 8y agoThis has bothered me for a long time. It looks like i'm seeing something other than what other people are seeing. So many of these documents just look awful. In this particular case it does not look good either: http://i.imgur.com/e11d0aK.png http://i.imgur.com/e11d0aK.png http://i.imgur.com/0Ysr7QQ.png http://i.imgur.com/0Ysr7QQ.png Look at the kerning on "Awk", it's not good. And look at the zoomed in version, the characters all have pixelisation and jaggies. These were just viewed using Firefox's default pdf viewer. Is there a way to view them and see better a quality version of the document?
- wernsey 8y agoI'll just share my biggest Awk-based project here: A tool that creates HTML documentation from Markdown comments in your source code. It does essentially what Javadoc does, except it uses Markdown to format text, which I find much more pleasing to the eyes when you read source code. The benefit of doing it in Awk is that if you want to use it in your project, you can just distribute a single script with your source code and add a two lines to your Makefile. Because of the ubiquity of Awk, you never have to worry whether people building your library has the correct tools installed. It doesn't have all the features that more sophisticated tools like Doxygen provides, but I'm going to keep on doing the documentation for my small hobby projects this way. [1] https://github.com/wernsey/d.awk https://github.com/wernsey/d.awk
- mauvehaus 8y agoIt is true that there exists an awk most everywhere, but I've been burned a few times by the discovery that nawk (Solaris (dating myself a bit there...)), gawk (many GNU/Linux distros), and mawk (some GNU/Linux distros? Don't know where I ran into it, but I have in the last couple of years) all have subtle incompatibilities around the edges. As I recall, gawk in particular has some extensions that are explicitly a superset of the standard funcionality. Which is great if you're only targeting gawk and know about it. It's less great if you think you're only targeting gawk, and discover later on that there's a system with a different awk you need to support. Your project looks neat, by the way. I'm looking forward to taking a closer look.
- wernsey 8y agoYes, there are differences. I used only standard awk facilities, and tested my project with gawk, mawk and nawk. I've also been burned when I discovered that Raspbian ships with an older version of mawk that did something differently in the way it processed regexen that caused my script to break.
- kazinator 8y agoOn the other hand, there is a POSIX standard, which you can use as a rough guide to write portable Awk if you suspect portable Awk is needed.
- gerbilly 8y agoIf you want a binary alternative to awk, try using lex.[1] You can feed in regexps and c code fragments and it will generate c code for you. [1] https://www.tldp.org/HOWTO/Lex-YACC-HOWTO-3.html https://www.tldp.org/HOWTO/Lex-YACC-HOWTO-3.html
- vasili111 8y agoLots of comments about awk, perl and sed for text proccessing. What about tcl?
- wenc 8y agoRecycling an older HN discussion on Awk vs Perl https://news.ycombinator.com/item?id=14647022 https://news.ycombinator.com/item?id=14647022
- 3uclid 8y agoFor context, I'm in university, but during one of my internships, a lot of the older developers always seemed to use awk/sed in really powerful ways. At the same time, I noticed a lot of the younger developers hardly used it. I'm not sure if it's a generational thing, but I thought that was interesting. Anyways, are there any good resources to learn awk/sed effectively?
- MikeTaylor 8y agoAwk really has been superseded by Perl (and therefore arguably by Python, Ruby, etc.) But sed remains a thing of beauty, all its own, and very well worth learning. Hardly a day goes by that I don't use it in some one-off command like for i in *.png; do pngtopnm < $i | cjpeg > `echo $i | sed 's/png$/jpeg/'`; done
- kazinator 8y agofor i in in *.png ; do pngtopnm $i | cjpeg > ${i#.png}.jpeg done
- ulzeraj 8y agoI was joking with my coworker some weeks ago about how awk is condemned to be forever used as a cut replacement.
- kazinator 8y agoTXR Lisp provides a Lisp-ified awk in a macro: http://www.nongnu.org/txr/txr-manpage.html#N-000264BC http://www.nongnu.org/txr/txr-manpage.html#N-000264BC > "Unlike Awk, the awk macro is a robust, self-contained language feature which can be used anywhere where a TXR Lisp expression is called for, cleanly nests with itself and can produce a return value when done. By contrast, a function in the Awk language, or an action body, cannot instantiate an local Awk processing machine. " The manual contains a translation of all of the Awk examples from the POSIX standard: http://www.nongnu.org/txr/txr-manpage.html#N-03D16283 http://www.nongnu.org/txr/txr-manpage.html#N-03D16283 The (-> name form ...) syntax above is scoped to the surrounding awk macro. Like in Awk, the redirection is identified by string. If multiple such expressions appear with the same name, they denote the same stream (within the lexical scope of the awk macro instance to which they belong). These are implicitly kept in a hash table. When the macro terminates (normally or via non-local jump like an exception), these streams are all closed.
- excitom 8y agoI always found it interesting that the Awk paradigm is also the basis for IBM's RPG language. Two very different environments coming up with basically the same elegant solution for the same problem: 1. Run zero or more setup operations. 2. Loop over the lines of a text file and process its columns into an output format. 3. Run zero or more cleanup operations at the end.
- mdavid626 8y agoI know many people will downvote, but in my opinion, just say no to this ancient "programming language". It's so confusing, completely text based, designed ages ago in an entirely different environment. There are many better alternatives, like Python or Powershell. Why not use them?
- dokem 8y agoThis pdf looks like a scanned book but I can highlight and copy text from it? What exactly is going on here? Does Chrome pdf viewer have built-in OCR?
- fulafel 8y agoPDF has long had this feature, most/all readers support it. There is a hidden textual representation included along with the scanned shown content.
- dokem 8y agoInteresting, why not just replace the text then? Where can I find more info about this? I was actually trying to find a good source of info about PDF's recently and couldn't really find much.
- romwell 8y ago>Interesting, why not just replace the text then? Why would one do that? It's destructive. A typeset book is much more than a text file. Not to mention OCR is not perfect, especially when math / special symbols are involved.
- fulafel 8y agoReplacing the typeset text with any reasonable fidelity seems like a much harder problem than reproducing the scan and providing the ocr'ed text content. It might still be a good idea to do, maybe some software does this. I don't have any references, sorry.
- JepZ 8y agoThat is a nice book. Starting with a practical tutorial and going into the structure and language features afterwards on a reasonable page count of just about 200 pages. I like to use awk when I need something a little more powerful than grep. Nevertheless, when I look at the examples and where the book is heading I prefer R for many of the tasks (in particular Rscript with a shebang). Just to give an example: If you have to manipulate a CSV file, that would most certainly be possible with awk, but some day there might be a record which does contain the separator and your program will produce some garbage. R on the other hand comes with sophisticated algorithms to handle CSV file correct. I truly respect awk for what it was and is but I also think that the use-cases where it is the best tool for the job has become very narrow over time.
- samuell 8y agoAs I do most of my daily work in cheminformatics with a (shell-based) workflow engine (http://scipipe.org http://scipipe.org), awk has turned out to be the perfect way of defining even quite complicated components based on just a shell command. These days, pretty much 50% of my components are CSV/TDV data munging with awk! :D (Can be hard to explain how this works without an image, so an (older) image is found in: https://twitter.com/smllmp/status/984173696448434176 https://twitter.com/smllmp/status/984173696448434176 )
- mbubb 8y agoThe lovely and humbling thing about this it was written 3 decades ago and the examples still work. Makes me think of another short elegant piece by Kenneth Church (?) called "Unix for Poets" which shows how to use core UNIX utils to work with text. Also from the mid to late 80s. Perl may have replaced sed and awk but they endure.
- thomastjeffery 8y agoFrom the introduction to Chapter 2: > Because it's a description of the complete language, the material is detailed, so we recommend that you skim it, then come back as necessary to check up on details. Any book that recommends skimming is doing something right.