14 ms·
Awk in 20 Minutes (2015)
- geocrasher 7y agoWhenever somebody says "Oh, I don't know how to use awk", this is the link I send them. It's easily the most useful tutorial for awk on the 'net. There's a lot more you can do with awk, but this site shows you the top percentage in a very short amount of time.
- dwoot 7y agoIt's crazy how this reappeared on HN only hours after I went searching for awk on Hacker News. Fred Hebert is so well known in the Erlang and Elixir communities. I have just coined this idea the "gateway drug" approach to learning new tools. We should strive to find those introductions that are small enough that someone can digest without a massive upfront time investment to get them past the front door :)
- dang 7y agoDiscussed at the time: https://news.ycombinator.com/item?id=8893302 https://news.ycombinator.com/item?id=8893302
- freedomben 7y agoThis is an excellent blog post. I will refer people to this! He touches on all of the important points. If you want a more thorough/deep exploration of Awk, I recently gave a talk on it (virtually) at Linux Fest Northwest (LFNW) 2020. Awk: Hack the planet['s text]! Part 1 (Presentation): https://www.youtube.com/watch?v=43BNFcOdBlY https://www.youtube.com/watch?v=43BNFcOdBlY Awk: Hack the planet['s text]! Part 2 (Exercises):https://www.youtube.com/watch?v=4UGLsRYDfo8 https://www.youtube.com/watch?v=4UGLsRYDfo8 If you want to try your hand at the exercises, they are on github: https://github.com/FreedomBen/awk-hack-the-planet https://github.com/FreedomBen/awk-hack-the-planet
- hackermailman 7y agoI learned awk from the chapter on filters in the pamphlet sized book 'unix programming environment' by Kernighan and Pike you can buy for $8
- combatentropy 7y agoI've heard many good things about that book, which is why I bought it several months ago. From the bits I have read, I like it. But I have not yet finished it, partly because it is not pamphlet sized. It is over 300 pages. But I look forward to the chapter on Filters.
- xwdv 7y agoEvery time I set out to use awk for some one time task I struggle to express what I need and end up saying fuck it and fire up vim and do exactly what I want with some quick macros. Am I the only one?
- yaktubi 7y agoI usually say duck it and dust off my shiny Perl necklace
- Noumenon72 7y agoI usually persevere and get it done in awk but then find the next time I need awk that I don't remember any of it and can barely understand my own examples.
- gabrielsroka 7y agobwk talking about awk, C, etc: https://youtu.be/Sg4U4r_AgJU https://youtu.be/Sg4U4r_AgJU
- aidenn0 7y agoThis is a great intro; I have the GNU Awk user's manual bookmarked because there are a lot of features in gawk you will only rarely use but are quite useful.
- bcyn 7y agoSo, I don't use awk right now. This answers "how to awk" more than "why use awk" for me (understandable given the 20 min claim). Does anyone have concrete, practical examples of use cases where awk made your life much easier?
- ggm 7y agoAwk '{ print $2 }' does LWSP gobbling.. cut -d' ' -f2 doesn't and this difference alone makes awk useful to me on a daily basis. Awk has a hash like perl which is very efficient. I use an awk expression to print uniq as they come in counted through the hash insert on new instead of uniq which prints at end. Awk count unique over 300,000,000 ips was as fast as perl and python and smaller memory footprint
- swimfar 7y agoLWSP=Linear White Space "Linear white space is: any number of spaces or horizontal tabs, and also newline (CRLF) if it is followed by at least one space or horizontal tab." [1] I don't know what LWSP gobbling is, though. Google didn't help. [1] https://stackoverflow.com/questions/21072713/what-exactly-is-the-linear-whitespace-lws-lwsp https://stackoverflow.com/questions/21072713/what-exactly-is...
- pstuart 7y agoIt means a contiguous collection of arbitrary white space characters are considered to be a single delimiter. With the parent example it means that column #2 could have any of number whitespaces between it and column #1.
- ggm 7y agoyes. whereas cut, considers ' ' (two spaces) as an instance of a blank column separated by the -d' ' space instance. Its like ,, in CSV being a blank column.
- skrebbel 7y ago
- spsrich2 7y agoit's amazingly useful. I have used it since 1987, most recently today.
- mauvehaus 7y agoI learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes with an ecosystem that is much larger still. I know I can use perl for what I use awk for, and when I've raised this point before, people have been quick to explain how to process input by lines, and conditionally do something. For basic stuff, the fact that that's basically all awk does[0] means there's so many fewer ways to do it wrong. I can't say the same of perl, even when I was more familiar with it. caveat: awk, nawk, mawk, and gawk don't necessarily share the same set of corner cases, and you may not get error messages that make sense to you when you bump into one with an unfamiliar awk. [0] I know, not really, and especially for gawk. I've written awk scripts a couple hundred lines in the past. It's true enough for the 20 minute version though. Edited for clarity.
- AceJohnny2 7y agoFor my & others' reference (because I keep having to look it up, just like parent): perl -ne "print $_;" '-n' will run the expression over each line of the input. This is the Awk-like mode. perl -pe "s/foo/bar/" '-p' will run the expression over each line of input, and print out the (possibly modified) line. This is the Sed-like mode. Slightly more info available in `man perlrun` or https://perldoc.perl.org/perlrun.html https://perldoc.perl.org/perlrun.html
- hf 7y agoOne particularly mnemonic collection of switches is 'plane': perl -plane 'my $script' which iterates over all files given on the command-line (or stdin) and + (p)rints every processed line back out + deals with (l)ine endings, in and out + (a)utosplits every line into @F I am aware that -n and -p are mutually exclusive, but as -p overrides -n, it's seems simpler to just keep 'plane' in mind and remove the 'p' if necessary.
- known 7y ago
- etxm 7y agoOne time I had a file and I needed all the values in a column and so I used: awk '{ print $2 }' my-file And it gave me what I wanted. It was cool.
- daenz 7y agoVery cool and I'm grateful for Awk to be presented in such a friendly way. However, at some point, it makes more sense to write your Awk script in something like Python, and my intuition says that that time is shortly after starting a basic Awk script. Using a real programming language is almost always the way to go with code that will become more complex over time (almost all code) and code you have to share with a team (learning curve).
- liammonahan 7y agoMy absolute favorite example of what's possible with awk is this[0] calculator from Ward Cunningham about splitting expenses on a ski trip. It's a really beautiful little piece of code well-adapted to this problem. [0] - https://c2.com/doc/expense/ https://c2.com/doc/expense/
- kqr 7y agoThis is the sort of thing us industry professionals need to study in more depth. It doesn't have the most flexible interface, but it's human-readable, and the implementation is simple, fast, and about as obviously correct as possible.
- cyrusmg 7y agoI am trying to parse the program based on the article above and comments on the link you posted. > The first occurrence of a variable name defines it as that sum. Subsequent occurrences become the stored value. With this quote in mind - why is second `SUM` reference replaced with `-138.95` and not `221.81` (stored value) ? EDIT: Never mind, now I see it's an exception on line 3.
- tekknolagi 7y agoThis reminds me of a story of a guy who wrote a whole company internal debit system like this (coffee, meals out, etc) and it turned into a currency. I feel like I either saw it here, or a similar forum. Anyone have a link? My searches have not found it...
- stuuuuuuuuu 7y agoThis one? https://royrapoport.blogspot.com/2011/05/coffee-and-its-effects-on-feature-creep.html https://royrapoport.blogspot.com/2011/05/coffee-and-its-effe...
- tekknolagi 7y agoThat's it! You found it! Thank you.
- known 7y agoCommon lines between 2 files awk 'NR==FNR{a[$0]; next} $0 in a' colors_1.txt colors_2.txt http://archive.vn/mmd80 http://archive.vn/mmd80
- mkl 7y agoThe link doesn't load for me, but that seems pretty unreadable. I'd probably sort, merge, and print duplicates: sort -m <(sort -u file1) <(sort -u file2) | uniq -d
- asicsp 7y agoyou can use comm as well # common lines comm -12 <(sort file1) <(sort file2) # lines unique to first file comm -23 <(sort file1) <(sort file2) # lines unique to second file comm -13 <(sort file1) <(sort file2) regarding readability, it is the same with any new tool or programming language, you'd need to be familiar with its syntax and idioms, someone not familiar with command line and sort/uniq commands will find your solution as alien as well
- asicsp 7y agoI've recently modified and published that tutorial as an ebook [1], which can be read from the github repo or downloaded as PDF (currently free). I am also updating the book to include exercises, other minor improvements as well as epub version - expected release by next weekend. [1] https://github.com/learnbyexample/learn_gnuawk https://github.com/learnbyexample/learn_gnuawk
- soufron 7y agoAwk in 2 minutes : google "awk this" and "awk that" :D
- j_z_reeves 7y agonice, I finally took the time to read the man pages for awk. And whipped out a script to count the number of errors occurred for a particular day for a postgres log file. cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort I just needed to know how awk programs are structured, the rest is just simple programming! EDIT: I'm not sure if it's actually correct however...
- j0057 7y agoWithout knowing the format of logfile, that still seems obviously correct to me.
- j_z_reeves 7y agoI was surprised by the counts[$1] = counts[$1] + 1, since I didn't think it would correctly coerce a non-existing value to a 0.
- xorcist 7y agoApart from the useless use of cat, since sort does the work here something like the following would probably suffice: grep ERROR logfile | cut -f 1 -d ' ' | sort | uniq -c
- j_z_reeves 7y agoyes that would also work! I forgot about the `-c` argument for uniq.
- j0057 7y agoThere's really nothing useless about that use of cat: it makes the pipeline compose better from left to right. It's not like you have to pay 25 cents for each process you spawn.
- xorcist 7y agoSo does the pipeline above. It's not detrimental to performance since an empty cat is a no-op in a pipeline. You can have any number of them. But commands should be written for humans to understand, and inserting no-ops is a distraction to the reader. In the trivial example, "grep needle haystack" reads better than "cat haystack | grep needle".
- Marcus316 7y agoI love using awk. It was fairly easy to pick up, and it slides into my command line workflow pretty well. That said, there's an amazing amount you can do with it, if you really try. Someone once joked that I should try building an IRC bot in AWK, so I did: https://github.com/Marcus316/rufus https://github.com/Marcus316/rufus There's no pactical reason for it, but it was fun to play with the idea.
- rafaele 7y agoGood tutorial. I like the concise survey of the language components. This is going to make working with awk a lot less awkward.
- arendtio 7y agoVery nice conclusion of the language. However, keep in mind that awk is not the fastest language around. A pro awk thread might not be the best place to tell this story but it is fresh and true: Last weekend I was playing around with some data. At first, I thought 'let's just write a line of awk and be done with it' and so I did. The execution took 20 seconds (about 17 million lines) and everything was fine. Later that day, I came across another task which seemed too complex for an awk one-liner so I took two lines of R and was surprised when R was done within 5 seconds on the same data set. I was happy because I found a faster tool than the one I had, but the lesson is, that just because you use a proven tool like awk, doesn't mean there aren't any better tools. Find out what works best for you.
- wodenokoto 7y agoWhat kind of task was it? I’d imagine loading a csv-like file and doing some somewhat heavy calculations, R would win, but when I think of AWK problems I think of text manipulation, and when I think of manipulating text I do not think of R Disclaimer: I don’t actually write any AWK, but learning it is on my bucket list.
- arendtio 7y agoThe tasks were different. AWK -> calculate the difference between every two lines: awk -F ',' 'NR!=1{printf "%.0f\n", $1-ll}NR==1{print ""}{ll=$1}' R -> Calculate the min, median, mean, and max: d<-scan("stdin", quiet=TRUE) cat(min(d), max(d), median(d), mean(d), sep=" ") AFAIK, calculating the difference to its previous line for every line should be faster for data that is not sorted. I didn't try to write that one in R though. I am sure there ways to make both things faster, I was just surprised as I didn't expect R to be faster with those two naive implementations.
- tannhaeuser 7y agoawk is a language with multiple implementations, so we can only talk about performance of a particular awk implementation. Which is especially true for awk because mawk (based on a vm rather than being a traditional interpreter) is so much faster than other awks if you can live with its limitations [1]. If you're using gawk, setting LANG=C also helps performance because no utf-8 honoring needs to be done. Speaking of which, I've noticed gawk on recent Ubuntus (19.10) appears broken and/or has regexp size limits breaking awk code with large regexpes (but haven't checked yet thorougly). [1]: https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-and-most-elegant-big-data-munging-language/ https://brenocon.com/blog/2009/09/dont-mawk-awk-the-fastest-...
- augustk 7y agoIt's also worth mentioning that local variables can be simulated using additional formal parameters. In AWK, any missing parameter in a function call is initialized to zero. Let's say we have a function CharCount which takes a character and a line of text and returns the number of occurrences of that character: function CharCount(ch, line, n) { ... } The line break in the parameter list is an AWK convention and indicates that n is a "local variable."
- ratsmack 7y agoI thought the convention was to separate local variables with three spaces like this: function CharCount(ch, line, n)
- js8 7y agoTo use Awk I tended, up on the ward I ended, it was awkward.
- jzer0cool 7y agosed grep & awk :)
- penguinjeff 7y agoGreat post. I didn't know Fred had a blog. Fred's writing style never ceases to amaze me. His book on property based testing in Elixir was a better introduction for me to Python's Hypothesis than any of the other tutorials I found online on the topic because it made me think about discovering properties of my code. Would recommend it to others.
- dmux 7y agoThe pattern-action paradigm is really simple to understand and I suspect it's what made Sinatra style web frameworks stand out.
- gumby 7y agoOne trick I use with awk, especially throwaway ones, is to use grep to subset the data before feeding it into awk. Often the case is using awk to poke at some data to diagnose a problem, not write a script to be run often or stuck in cron. So to use an example from this nice short article: I might do grep GET log-file | awk blah-blah. Then awk doesn’t need to consider the lines I don’t care about. This is especially useful when iteratively writing the awk script.
- yiyus 7y agoI find easier to do awk /GET/{blah-blah}
- gumby 7y agoOh yeah, sure! I should have been clear that when dredging a really large log file or such it can be faster to run awk on a much smaller subset when your awk program has a lot of productions. I find it easier to think "OK, just run awk on these lines that matter." Six of one half dozen of the other.
- alexhutcheson 7y agoThere are two use-cases I’ve run into where awk really shines, and is hard to replace: 1. Writing scripts for environments that only have Busybox. Technically you can write scripts in ash, but I don’t recommend it for anything beyond a couple lines. It’s missing a lot of the features from Bash that make scripting easier, and it’s easy to get mixed up if you’re used to Bash and write things that don’t work. Awk is the best scripting language available, even if you’re doing things that don’t exactly match what it was designed to do. 2. Snippets that are meant to be copy+pasted from documentation or how-to articles. In that case, it’s often not easy to distribute a separate script file, so a CLI “one-liner” is preferred. You also can’t count on Perl, Python, etc. being available on the user’s system, but awk is pretty universal. For most other cases, I tend to create a new .py file and write a quick Python script. Even if it’s a little more overhead, it helps keep my Python skills sharp, and often it turns out that what I actually want is a little more complicated than my initial idea anyway.
- dwheeler 7y agoNice intro. Awk is a useful tool when you want to do simple line-by-line processing. Note: The article says that awk patterns can't capture groups. The standard doesn't provide that functionality, but if you use the widely-available gawk implementation, gawk does have that capability (use "match").
- globular-toast 7y agoI recommend this article to people all the time and often come back to it myself. I don't use awk as much these days but when I worked in bioinformatics I was fluent and loved it.
- blackrock 7y agoWhat’s better than awk, are one-liner python programs. Then, you can even alias it.
- jftuga 7y agoOne of the things I like to do with mawk.exe under Windows is to automate the same command over a group of files. Let's say the myPgm only takes one file name as a command line parameter, then I can so something like this: dir *.xyz /s/b | mawk "{print 'myPgm -x '$0}" | cmd If the file paths have spaces in them, then you have to wrap the name within double quotes. I have found it challenging to output those in mawk -- at least easily any way. In this case, I have a windows version of tr.exe In this case, I can do something like this: dir *.xyz /s/b | mawk "{print 'myPgm -x ~'$0'~'}" | tr ~ \042 | cmd Although somewhat crude, it is effective.
- known 7y ago#to print duplicate lines awk '++seen[$0] > 1' filename.txt
- kazinator 7y agoBy the way, Awk is whitespace sensitive: pattern { action } # valid pattern { # valid action } pattern # not what you think { action } # this action has no pattern There cannot be a newline between pattern an action. This feature allows either the pattern or the action to be omitted without ambiguity. pattern # pattern with default { print } action { action } # unconditional action pattern { # pattern with action action } Items can be put on a single line without ambiguity using semicolons: pattern ; { action } ; pattern { action } A pattern can have multiple patterns separated by a comma. That syntax admits optional line separation after the comma separators: pattern, pattern, pattern { action } The action fires by a match for any of the patterns. The POSIX standard Awk expression grammar has no comma operator, on the other hand; the comma exists only for separating patterns, and function arguments/parameters, and in the print statement syntax. Plus, JNIL (just now I learned). The in operator allows comma-separated expressions for testing "multi-dimensional" array membership. Here is a hello, world: $ awk 'BEGIN { a[1,2,3] = 4 ; print (1,2,3) in a }' 1 (Note that multi-dimensional arrays in Awk are simulated; a string index is generated for that 1,2,3).
- fithisux 7y agoThis is the manual I needed the last 20 years.
- marcinzelent 7y agoLast year, I attended a tech event where Rasmus Lerdorf (the creator of PHP) was a guest speaker. When asked about his favorite programming language, he said: AWK. He didn't elaborate why but looking at this article, I think there is something to it.
- alexerrrr123 7y agoLa blah blah by Lala blah Florida
- machinesbuddy 7y agoCould remove line 32 and just put `{ flag = 0 }` before line 31, right?