3 ms·

Hmm. I'm sure this question will induce a flamewar of practical "#NeverAwk"-ers fighting toolbelt bloat, versus tech-hoarding AWK apologists arguing against thr

by Chris2048 7y ago

Hmm. I'm sure this question will induce a flamewar of practical "#NeverAwk"-ers fighting toolbelt bloat, versus tech-hoarding AWK apologists arguing against throwing something out given if fills <niche>. Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from actual common practise, and or solid comparable benchmarks. Show me a range of practical examples, for each competing env (awk, sed, perl, python, ruby, maybe bash). Include: - time it takes to teach a total novice (the time it takes to learn whatever is needed that example, not the entire language) - how easy it is to recall said knowledge at a later date - how fast example is multiplied by how much you are likely to use it = actual time saved in terms of execution. for small, fast examples the difference is irrelevant, a 10x speedup that is 0.1s vs 0.01s is meaningless. - how extendable an example is. Hence the original example should include a series of extensions to the original task, to demonstrate how flexible / composable they are: e.g. task 1) count lines in a file; task 2) count lines in a file, then add 42 to it; I suspect awk falls behind in practicality vs perl (which can do simple one-liners, but also more complicated constructs), but perhaps has a hidden virtue wrt speed in more expensive tasks, ala https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841 or https://news.ycombinator.com/item?id=20293579 https://news.ycombinator.com/item?id=20293579

arminiusreturns 7y ago
I like to tell the story of when I was doing some genetics data wrangling and spent three days writing some perl code and I kept failing, then sent one of the researchers an email and he suggested an awk method and I turned 3 pages of perl into an awk one liner that works just fine. Now, its probably because I don't know perl very well, but as an ops type who doesnt have the classical dev/cs education, tools like it are my goto.
- bonoboTP 7y ago
  I find Python list/dict comprehensions, zip, range, enumerate etc. and the itertools module very good for such things. And you have immediate access to so many useful modules (csv, json, xml) and can easily extract code fragments into functions. You can also execute shell-like commands with subprocess.check_output() without ever worrying again about escaping strings or accidentally splitting them at spaces or whatever. Clever one liners are difficult to comprehend. It's better to break them up to a few variable assignments with descriptive, long names without abbreviations.
  - arminiusreturns 7y ago
    I agree python seems to have taken over in this space since then (7+ yrs ago) and is usually a better tool in bioinformatics, I was just using the tools I knew as an ops dude.
jandrese 7y ago
I sat down and read through the Awk manual once just to learn and was pleasantly surprised at how orthogonal the language is and how much it offers. The only problem is that it was written by people using ed on PDP computers and that kind of shows. The primary logic is "filter out lines X and apply transform Y" is completely natural to someone using an editor like ed, but is fairly foreign to modern computer users. Most people aren't going to take the time to learn an obscure commandline tool these days, especially since it comes from the Dennis Ritchie school of "errors are like angry housewives, you know what you did" debugging.
flukus 7y ago
> Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from actual common practise, and or solid comparable benchmarks. I think your missing the point of awk. The O'Reilly sed and awk book has some complex examples, but when I look at my own usage they are all toy examples within a much larger scope. It's more like a special DSL extension for my shell than something I'd pick to build the entire solution, so a comparison to perl, python and ruby don't really make sense, they are general purpose languages but awk just has a couple of features that make it a very specialized yet useful one. As an example, a have a system for importing and parsing log files that mostly done from a shell script, awk is used in two parts. The first is to transform a structured and easy to read file (records '\n\n' separated) into a csv easier to consume for bash, there's probably quite a few options to do this from tr to bash and it's done inline. The second is to filter the results down to what I need, so I have scripts like: #!/usr/bin/awk -f /some common error I don't care about/ { next } #skip line /other common error/ { next } /Error/ { print $0 } #this prints error lines, alternatively: /Error/ && !errors[gensub($1", "", "g", $0)]++ { print $0 } #print each error once {next} #skip everything else Apart from one single line which wasn't in the original that's something you could teach a total novice in minutes, the /pattern/{action} syntax is about as simple as programming can be. Execution speed could probably be improved with a specific program but I suspect the bottleneck would be the spinning disk anyway, I run this over hundreds of MB every few minutes and it's not a problem, when I run it manually it's near instantaneous, I spend longer waiting for the desktop calculator to open up these days.
- Chris2048 7y ago
  Can you give an example of when you'd choose AWK over perl? perl is general purpose, but that doesn't mean it can't be used for one-liners.

arminiusreturns 7y ago

I like to tell the story of when I was doing some genetics data wrangling and spent three days writing some perl code and I kept failing, then sent one of the researchers an email and he suggested an awk method and I turned 3 pages of perl into an awk one liner that works just fine. Now, its probably because I don't know perl very well, but as an ops type who doesnt have the classical dev/cs education, tools like it are my goto.

bonoboTP 7y ago
I find Python list/dict comprehensions, zip, range, enumerate etc. and the itertools module very good for such things. And you have immediate access to so many useful modules (csv, json, xml) and can easily extract code fragments into functions. You can also execute shell-like commands with subprocess.check_output() without ever worrying again about escaping strings or accidentally splitting them at spaces or whatever. Clever one liners are difficult to comprehend. It's better to break them up to a few variable assignments with descriptive, long names without abbreviations.
- arminiusreturns 7y ago
  I agree python seems to have taken over in this space since then (7+ yrs ago) and is usually a better tool in bioinformatics, I was just using the tools I knew as an ops dude.

jandrese 7y ago

I sat down and read through the Awk manual once just to learn and was pleasantly surprised at how orthogonal the language is and how much it offers. The only problem is that it was written by people using ed on PDP computers and that kind of shows. The primary logic is "filter out lines X and apply transform Y" is completely natural to someone using an editor like ed, but is fairly foreign to modern computer users. Most people aren't going to take the time to learn an obscure commandline tool these days, especially since it comes from the Dennis Ritchie school of "errors are like angry housewives, you know what you did" debugging.

flukus 7y ago

> Here's the thing: these arguments all too commonly focus on subjective notions of "simplicity", and toy examples divorced from actual common practise, and or solid comparable benchmarks. I think your missing the point of awk. The O'Reilly sed and awk book has some complex examples, but when I look at my own usage they are all toy examples within a much larger scope. It's more like a special DSL extension for my shell than something I'd pick to build the entire solution, so a comparison to perl, python and ruby don't really make sense, they are general purpose languages but awk just has a couple of features that make it a very specialized yet useful one. As an example, a have a system for importing and parsing log files that mostly done from a shell script, awk is used in two parts. The first is to transform a structured and easy to read file (records '\n\n' separated) into a csv easier to consume for bash, there's probably quite a few options to do this from tr to bash and it's done inline. The second is to filter the results down to what I need, so I have scripts like: #!/usr/bin/awk -f /some common error I don't care about/ { next } #skip line /other common error/ { next } /Error/ { print $0 } #this prints error lines, alternatively: /Error/ && !errors[gensub($1", "", "g", $0)]++ { print $0 } #print each error once {next} #skip everything else Apart from one single line which wasn't in the original that's something you could teach a total novice in minutes, the /pattern/{action} syntax is about as simple as programming can be. Execution speed could probably be improved with a specific program but I suspect the bottleneck would be the spinning disk anyway, I run this over hundreds of MB every few minutes and it's not a problem, when I run it manually it's near instantaneous, I spend longer waiting for the desktop calculator to open up these days.

Chris2048 7y ago
Can you give an example of when you'd choose AWK over perl? perl is general purpose, but that doesn't mean it can't be used for one-liners.