4 ms·

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocatio

by oefrha 6y ago

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'

mellow-lake-day 6y ago
Speaking of AWK, is there a more succinct version of the AWK programming language book that I can use for learning? https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...
- sn41 6y ago
  You can use the notes of Bruce Barnett for quick lookups: https://www.grymoire.com/Unix/Awk.html https://www.grymoire.com/Unix/Awk.html
- asicsp 6y ago
  I wrote a book on GNU awk one-liners: https://learnbyexample.github.io/learn_gnuawk/ https://learnbyexample.github.io/learn_gnuawk/ There's plenty of examples and exercises, plus an entire chapter dedicated to regular expressions.
- dredmorbius 6y ago
  GNU AWK User's Guide is reasonably succinct, while being fairly comprehensive. https://www.gnu.org/software/gawk/manual/gawk.html https://www.gnu.org/software/gawk/manual/gawk.html The manpage alone is quite useful, though lacks some useful details. The awk FAQ provides general background, including diffeerences between implementations: http://www.faqs.org/faqs/computer-lang/awk/faq/ http://www.faqs.org/faqs/computer-lang/awk/faq/
- boneitis 6y ago
  On mobile; pardon the link format. https://m.youtube.com/watch?v=43BNFcOdBlY https://m.youtube.com/watch?v=43BNFcOdBlY In an afternoon, you can learn enough to be more than dangerous. The presenter drafted up a set of (easy, yet practical) companion exercises and a pt2 video with his own answers. I cannot recommend this talk enough. Although I hadn't the opportunity to put the knowledge to use and have since forgotten it (i don't work in tech), i find comfort in knowing i can reacquire the power of awk even sooner than the already-short first time around.
dredmorbius 6y ago
Or do the case-conversion, minimum word-length restriction, and counting in awk itself, here using associative arrays: awk < file ' { $0=tolower($0); # lowercase conversion gsub( "[^a-z ]", "", $0); # remove non-alpha + space chars for(i=1;i<=NF;i++0) { # increment word count if word length > 2 if(length($i)>2) count[$i]++ } }; END{ # report for(word in count) printf( "%-20s %i\n", word, count[word]) } ' | sort -k2nr -k1 This omits the header (can be trivially added). The external sort can be internalised in gawk using asort(), or by printing to sort via a command: cmd="print -2kr -k1" for<loop> printf( <args> ) | cmd close(cmd)
- derefr 6y ago
  When you have such a large awk script, you may as well also demonstrate that Awk is a “real scripting language” by putting it into its own script with a `#!/usr/bin/awk -f` shebang line. Much easier to edit than trying to get your shell to do it. (Unless you’re using a fancy editing shell like Emacs’ shell mode; but at that point, why would you need Awk? You probably know far better how to use your own shell-runtime’s text-manipulation primitives.)
  - dredmorbius 6y ago
    Sure. Though such scripts often begin as shell one-liners. This one did. Never left it, as it happens. C-x C-e FTFW. (I expanded, indented, and commented the example for HN.)
- hnlmorg 6y ago
  If you're going to those lengths then you might as well write it in Go or Python and get the performance boost on larger datasets. However the point often missed when people see awk piped into other CLI tools is that code is intentionally optimised for one time writing rather than maximising awk's usefulness. It's the same with the GPs comment too. In that example the awk code was longer than the awk|sed equivalent. Don't get me wrong, I am a big fan of awk myself. But sometimes people get so hung up on better usage of awk that they lose sight of the point behind the command line example.
jhoechtl 6y ago
I beg to differ. I did a lot of csv wrangling on Unix. Csv is a beast. My tools of choice ultimately was miller, an absolutely underrated tool: https://github.com/johnkerl/miller https://github.com/johnkerl/miller
- nicoburns 6y ago
  Oo, I've not come across this one before. Looks useful! In a similar vein of CSV tools that seem to be underrated, I recently came across Daff which does excellent tabular diffs. https://github.com/paulfitz/daff https://github.com/paulfitz/daff
- leonim 6y ago
  Totally agree. I think mlr is a wonderful CLI tool. It is very robust, and can handle/convert multiple tabular formats including csv, tsv, json, fixed-format, etc. It has a pretty decent text output formatting, with the --opprint flag. I use to be very comfortable using awk/sed/perl/sort/uniq/tr/tail/head from the CLI for the sort of data cleaning this article is talking about. However, over the past year I've found I use VisiData https://github.com/saulpw/visidata https://github.com/saulpw/visidata for interactive work. If I need to clean up the data first, I'll use mlr or jq as input to Visidata. If my data is too dirty for mlr, then I'll use Unix toolbox tools mentioned as input to mlr, jq or VisiData. VisiData provides some ability to script, but when possible I prefer to have the shell do the scripting with all the tools mentioned as input to Visidata.

by oefrha 6y ago

mellow-lake-day 6y ago

Speaking of AWK, is there a more succinct version of the AWK programming language book that I can use for learning? https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

sn41 6y ago
You can use the notes of Bruce Barnett for quick lookups: https://www.grymoire.com/Unix/Awk.html https://www.grymoire.com/Unix/Awk.html
asicsp 6y ago
I wrote a book on GNU awk one-liners: https://learnbyexample.github.io/learn_gnuawk/ https://learnbyexample.github.io/learn_gnuawk/ There's plenty of examples and exercises, plus an entire chapter dedicated to regular expressions.
dredmorbius 6y ago
GNU AWK User's Guide is reasonably succinct, while being fairly comprehensive. https://www.gnu.org/software/gawk/manual/gawk.html https://www.gnu.org/software/gawk/manual/gawk.html The manpage alone is quite useful, though lacks some useful details. The awk FAQ provides general background, including diffeerences between implementations: http://www.faqs.org/faqs/computer-lang/awk/faq/ http://www.faqs.org/faqs/computer-lang/awk/faq/
boneitis 6y ago
On mobile; pardon the link format. https://m.youtube.com/watch?v=43BNFcOdBlY https://m.youtube.com/watch?v=43BNFcOdBlY In an afternoon, you can learn enough to be more than dangerous. The presenter drafted up a set of (easy, yet practical) companion exercises and a pt2 video with his own answers. I cannot recommend this talk enough. Although I hadn't the opportunity to put the knowledge to use and have since forgotten it (i don't work in tech), i find comfort in knowing i can reacquire the power of awk even sooner than the already-short first time around.

dredmorbius 6y ago

Or do the case-conversion, minimum word-length restriction, and counting in awk itself, here using associative arrays: awk < file ' { $0=tolower($0); # lowercase conversion gsub( "[^a-z ]", "", $0); # remove non-alpha + space chars for(i=1;i<=NF;i++0) { # increment word count if word length > 2 if(length($i)>2) count[$i]++ } }; END{ # report for(word in count) printf( "%-20s %i\n", word, count[word]) } ' | sort -k2nr -k1 This omits the header (can be trivially added). The external sort can be internalised in gawk using asort(), or by printing to sort via a command: cmd="print -2kr -k1" for<loop> printf( <args> ) | cmd close(cmd)

derefr 6y ago
When you have such a large awk script, you may as well also demonstrate that Awk is a “real scripting language” by putting it into its own script with a `#!/usr/bin/awk -f` shebang line. Much easier to edit than trying to get your shell to do it. (Unless you’re using a fancy editing shell like Emacs’ shell mode; but at that point, why would you need Awk? You probably know far better how to use your own shell-runtime’s text-manipulation primitives.)
- dredmorbius 6y ago
  Sure. Though such scripts often begin as shell one-liners. This one did. Never left it, as it happens. C-x C-e FTFW. (I expanded, indented, and commented the example for HN.)
hnlmorg 6y ago
If you're going to those lengths then you might as well write it in Go or Python and get the performance boost on larger datasets. However the point often missed when people see awk piped into other CLI tools is that code is intentionally optimised for one time writing rather than maximising awk's usefulness. It's the same with the GPs comment too. In that example the awk code was longer than the awk|sed equivalent. Don't get me wrong, I am a big fan of awk myself. But sometimes people get so hung up on better usage of awk that they lose sight of the point behind the command line example.

jhoechtl 6y ago

I beg to differ. I did a lot of csv wrangling on Unix. Csv is a beast. My tools of choice ultimately was miller, an absolutely underrated tool: https://github.com/johnkerl/miller https://github.com/johnkerl/miller

nicoburns 6y ago
Oo, I've not come across this one before. Looks useful! In a similar vein of CSV tools that seem to be underrated, I recently came across Daff which does excellent tabular diffs. https://github.com/paulfitz/daff https://github.com/paulfitz/daff
leonim 6y ago
Totally agree. I think mlr is a wonderful CLI tool. It is very robust, and can handle/convert multiple tabular formats including csv, tsv, json, fixed-format, etc. It has a pretty decent text output formatting, with the --opprint flag. I use to be very comfortable using awk/sed/perl/sort/uniq/tr/tail/head from the CLI for the sort of data cleaning this article is talking about. However, over the past year I've found I use VisiData https://github.com/saulpw/visidata https://github.com/saulpw/visidata for interactive work. If I need to clean up the data first, I'll use mlr or jq as input to Visidata. If my data is too dirty for mlr, then I'll use Unix toolbox tools mentioned as input to mlr, jq or VisiData. VisiData provides some ability to script, but when possible I prefer to have the shell do the scripting with all the tools mentioned as input to Visidata.