27 ms·

I consider awk to be the most useful and underused language in the UNIX ecosystem. I use it daily to analyze, transform, and assemble data, and it always blows

I consider awk to be the most useful and underused language in the UNIX ecosystem. I use it daily to analyze, transform, and assemble data, and it always blows my mind that so few people really know how to use it at a decent level. This is an excellent book to give a real idea of what awk is capable of.

taeric 8y ago
I think my hiccup is that it seems like good awk programmers are expected to be able to pass the full bloody program in as a simple string argument in a bash one-liner. Which is clearly not tenable. There are a few things I have done which felt more difficult than they should. However, the vast majority of dealing with structured data using awk really has been a pleasant surprise.
- asicsp 8y ago
  one-liners are fun though, especially if you like code golfing.. if you're interested, you could give my tutorial(https://github.com/learnbyexample/Command-line-text-processing/blob/master/gnu_awk.md https://github.com/learnbyexample/Command-line-text-processi...) a try it has over 300 examples..
  - ageofwant 8y ago
    Thanks for that mate. I'm a regular customer your work is greatly appreciated.
    - asicsp 8y ago
      thanks for the feedback :) happy learning
  - natecavanaugh 8y ago
    I gotta say, I really enjoyed skimming through this, and saving it for future reference. I can already tell there are some basic things that I should've memorized with all the one liners I've used, but because most were cobbled together for the immediate task (and many online forum searches assume a lot of prior knowledge), so t this presentation, and cookbook style references really help me hold the practical benefits in memory alot longer, so thanks again!
    - asicsp 8y ago
      thanks, that means a lot to me :) if you are interested in text processing in general, do check the other chapters for grep/sed/perl/sort/paste/pr/etc I learned a lot writing them up and marvel at the speed and functionality provided by these tools.. for many cases, it is lot easier to write a bash script combining these tools than to write a Perl/Python/Ruby script..
  - xpil 8y ago
    Ahh, what a useful link! Bookmarking immediately. I use AWK like once a year but when I do, I usually struggle a lot. Your examples are really helpful!
    - asicsp 8y ago
      thanks, it's a reference for me as well - not easy to remember all the features and tricks
- schoen 8y ago
  I think some of the examples in popular documentation use awk -f to read a program from a file, so I don't feel like that's regarded as an unreasonable way to run awk code. It's true that many programs might well just have a single action (or a single action plus a BEGIN and END), but I'm not sure that resorting to awk -f will give great offense.
- coliveira 8y ago
  Awk makes it easy to write one liners, but it is also very easy to save your script to a file and run it with the -f option.
- Stratoscope 8y ago
  Awk did have this reputation of hard-to-read one liners, but it was also always possible to write readable code in it much as you would do in any language. My favorite bit of Awk code that I ever wrote was this print formatting program for the HP LaserJet II: https://github.com/geary/awk/blob/master/LJPII.AWK https://github.com/geary/awk/blob/master/LJPII.AWK It printed source code in a "2-up" format, with two pages printed side by side in landscape mode. It looked for the kind of separators that I and many people were fond of back in those days (the source code is a good example) and converted those to graphic boxes. And it tried to be smart about things like breaking pages before a comment box, not after. I wish I still have a sample printout handy. For 300 lines of what I thought was fairly cleanly written Awk code, it made some nicely formatted printouts.
  - oblio 8y ago
    That's very clean looking code and I think it would be considered so regardless of the language it was written in, IMHO. Nice! :)
- pmarin 8y ago
  what I usually do is to wrap the awk script in to a sh function. I never do one-liners unless it is a well known idiom.
ajross 8y ago
> it always blows my mind that so few people really know how to use it at a decent level Not nearly as surprising as it is to me that now that most developers have forgotten perl, they're turning to awk as an inspiring example of a bygone era. Seriously: perl basically replaced awk in the mid-90's. It absorbed all the great lessons and added two or three dozen innovations. But it had scary syntax, so everyone used python (which did not replace awk very effectively) and forgot perl. So now we're back at awk. And the scary syntax is all in the Rust world.
- kqr 8y ago
  What I like about AWK is that it is described by a few pages of the POSIX standard. If Perl was that ubiquitous and simple, I would prefer it over AWK.
  - ajross 8y ago
    To be clear: I like that about it too. But in the real world, we want tooling that does more stuff. In the mid-90's, "everyone" knew perl, because it was the best choice for problems in this space. And in a world where everyone knows perl, there is no place for awk. But now we live in a world where no one knows perl, and languages like python and javascript are clumsy and weird in this space. And that makes awk look clever and elegant. All I'm saying is that in perl-world (which was a real place!), awk wasn't clever and elegant, it was stale and primitive. And that to me is more surprising than awk's cleverness.
    - fiddlerwoaroof 8y ago
      I think people underestimate the importance of medium-powered tools. Perl is great, but awks limitations make it easier to write in a way that the next person can maintain.
      - fiddlerwoaroof 8y ago
        That being said, the couple times I’ve paired with a perl wizard were revelatory.
deleted 8y ago
[deleted]
busterarm 8y ago
I think my tendency to reach for grep/sed/awk might be the thing that led me away from software development full-time and into the ops world. Too many times I encountered 200+ line Python scripts that I could replace with an Awk one-liner and a cron job. My lazy habit in my Ruby work eventually became to parse JSON/XML/whatever into usable text, shelling out to sed/awk and working with the results. This would not only save me LoC, but was less error-prone.
boobsbr 8y ago
I never got past using awk for string replacement that I couldn't pull off as a regex in grep...
deleted 8y ago
[deleted]
Annatar 8y ago
I use AWK daily; it’s my workhorse programming language and automation tool. Small, fast, no dependencies and portable. Extremely powerful programming language for big data processing and data driven programming, I often use it to generate shell scripts based on some arbitrary data. With no memory management and providng hash arrays, AWK is an absolute delight to program in. Using it with the functional programming paradigm makes it even more powerful. It blows my mind that Aho, Weinberger and Kernighan managed to design something so versatile and yet so small and fast. I’ve also been using it to replace Python scripts with a ratio of Python:AWK being anywhere from 3:1 to 10:1 as far as lines of code needed to get the same tasks done.
gnuvince 8y ago
Agreed. I learned awk a bit late in my Linux journey (started using Linux in 1999, learned awk in 2016), but I can't imagine not using it daily now. Great tool to manipulate data, and it's great for when you want a Unix tool that doesn't exist (e.g., computing the sum or mean of a stream of numbers).
mdip 8y ago
it always blows my mind that so few people really know how to use it at a decent level I'll second this -- it's by far the most used swiss-army knife in my arsenal[0]. A little part of me dies when I see scripts doing grep|awk, using awk simply to print a column. After taking some time to actually read the gawk user manual a few years ago, I've found that I can do most things in awk that I was previously using grep/sed (and to a lesser extent) perl/python for in the shell. In my environment, it also comes with the benefit that I know the awk code I'm writing is supported by the version of gawk that's installed on every server/computer I come into contact with -- no having to ensure the right python/perl is available. One thing I wish there was better documentation on, though, is creating complete gawk scripts. With @include directives coupled with AWK_PATH, it's really convenient to build up a handful of utility functions that do frequent things and I've found, on several occasions, that I end up writing an .awk script instead of a bash/zsh script and it ends up being a much more straight-forward set of code. [0] Well, specifically, the 'gawk' variant
- ausjke 8y ago
  why not write something about the @include directives like a blog or some posts, that will be awesome.
wernsey 8y ago
One thing I also really like about Awk is that you can keep 99% of the language in your head. With Awk it is seldom necessary to even reference the man page. I don't have to interrupt myself with looking up the syntax or the order of the parameters to a function.
gitgacutils 8y ago
awk is a fundamental part of the unix ecosystem. It's in the hall of fame of utils along with the likes of grep. If you use unix or linux in any semi-serious capacity, you are familiar with awk.

taeric 8y ago

I think my hiccup is that it seems like good awk programmers are expected to be able to pass the full bloody program in as a simple string argument in a bash one-liner. Which is clearly not tenable. There are a few things I have done which felt more difficult than they should. However, the vast majority of dealing with structured data using awk really has been a pleasant surprise.

asicsp 8y ago
one-liners are fun though, especially if you like code golfing.. if you're interested, you could give my tutorial(https://github.com/learnbyexample/Command-line-text-processing/blob/master/gnu_awk.md https://github.com/learnbyexample/Command-line-text-processi...) a try it has over 300 examples..
- ageofwant 8y ago
  Thanks for that mate. I'm a regular customer your work is greatly appreciated.
  - asicsp 8y ago
    thanks for the feedback :) happy learning
- natecavanaugh 8y ago
  I gotta say, I really enjoyed skimming through this, and saving it for future reference. I can already tell there are some basic things that I should've memorized with all the one liners I've used, but because most were cobbled together for the immediate task (and many online forum searches assume a lot of prior knowledge), so t this presentation, and cookbook style references really help me hold the practical benefits in memory alot longer, so thanks again!
  - asicsp 8y ago
    thanks, that means a lot to me :) if you are interested in text processing in general, do check the other chapters for grep/sed/perl/sort/paste/pr/etc I learned a lot writing them up and marvel at the speed and functionality provided by these tools.. for many cases, it is lot easier to write a bash script combining these tools than to write a Perl/Python/Ruby script..
- xpil 8y ago
  Ahh, what a useful link! Bookmarking immediately. I use AWK like once a year but when I do, I usually struggle a lot. Your examples are really helpful!
  - asicsp 8y ago
    thanks, it's a reference for me as well - not easy to remember all the features and tricks
schoen 8y ago
I think some of the examples in popular documentation use awk -f to read a program from a file, so I don't feel like that's regarded as an unreasonable way to run awk code. It's true that many programs might well just have a single action (or a single action plus a BEGIN and END), but I'm not sure that resorting to awk -f will give great offense.
coliveira 8y ago
Awk makes it easy to write one liners, but it is also very easy to save your script to a file and run it with the -f option.
Stratoscope 8y ago
Awk did have this reputation of hard-to-read one liners, but it was also always possible to write readable code in it much as you would do in any language. My favorite bit of Awk code that I ever wrote was this print formatting program for the HP LaserJet II: https://github.com/geary/awk/blob/master/LJPII.AWK https://github.com/geary/awk/blob/master/LJPII.AWK It printed source code in a "2-up" format, with two pages printed side by side in landscape mode. It looked for the kind of separators that I and many people were fond of back in those days (the source code is a good example) and converted those to graphic boxes. And it tried to be smart about things like breaking pages before a comment box, not after. I wish I still have a sample printout handy. For 300 lines of what I thought was fairly cleanly written Awk code, it made some nicely formatted printouts.
- oblio 8y ago
  That's very clean looking code and I think it would be considered so regardless of the language it was written in, IMHO. Nice! :)
pmarin 8y ago
what I usually do is to wrap the awk script in to a sh function. I never do one-liners unless it is a well known idiom.

ajross 8y ago

> it always blows my mind that so few people really know how to use it at a decent level Not nearly as surprising as it is to me that now that most developers have forgotten perl, they're turning to awk as an inspiring example of a bygone era. Seriously: perl basically replaced awk in the mid-90's. It absorbed all the great lessons and added two or three dozen innovations. But it had scary syntax, so everyone used python (which did not replace awk very effectively) and forgot perl. So now we're back at awk. And the scary syntax is all in the Rust world.

kqr 8y ago
What I like about AWK is that it is described by a few pages of the POSIX standard. If Perl was that ubiquitous and simple, I would prefer it over AWK.
- ajross 8y ago
  To be clear: I like that about it too. But in the real world, we want tooling that does more stuff. In the mid-90's, "everyone" knew perl, because it was the best choice for problems in this space. And in a world where everyone knows perl, there is no place for awk. But now we live in a world where no one knows perl, and languages like python and javascript are clumsy and weird in this space. And that makes awk look clever and elegant. All I'm saying is that in perl-world (which was a real place!), awk wasn't clever and elegant, it was stale and primitive. And that to me is more surprising than awk's cleverness.
  - fiddlerwoaroof 8y ago
    I think people underestimate the importance of medium-powered tools. Perl is great, but awks limitations make it easier to write in a way that the next person can maintain.
    - fiddlerwoaroof 8y ago
      That being said, the couple times I’ve paired with a perl wizard were revelatory.

deleted 8y ago

[deleted]

busterarm 8y ago

I think my tendency to reach for grep/sed/awk might be the thing that led me away from software development full-time and into the ops world. Too many times I encountered 200+ line Python scripts that I could replace with an Awk one-liner and a cron job. My lazy habit in my Ruby work eventually became to parse JSON/XML/whatever into usable text, shelling out to sed/awk and working with the results. This would not only save me LoC, but was less error-prone.

boobsbr 8y ago

I never got past using awk for string replacement that I couldn't pull off as a regex in grep...

deleted 8y ago

[deleted]

Annatar 8y ago

I use AWK daily; it’s my workhorse programming language and automation tool. Small, fast, no dependencies and portable. Extremely powerful programming language for big data processing and data driven programming, I often use it to generate shell scripts based on some arbitrary data. With no memory management and providng hash arrays, AWK is an absolute delight to program in. Using it with the functional programming paradigm makes it even more powerful. It blows my mind that Aho, Weinberger and Kernighan managed to design something so versatile and yet so small and fast. I’ve also been using it to replace Python scripts with a ratio of Python:AWK being anywhere from 3:1 to 10:1 as far as lines of code needed to get the same tasks done.

gnuvince 8y ago

Agreed. I learned awk a bit late in my Linux journey (started using Linux in 1999, learned awk in 2016), but I can't imagine not using it daily now. Great tool to manipulate data, and it's great for when you want a Unix tool that doesn't exist (e.g., computing the sum or mean of a stream of numbers).

mdip 8y ago

it always blows my mind that so few people really know how to use it at a decent level I'll second this -- it's by far the most used swiss-army knife in my arsenal[0]. A little part of me dies when I see scripts doing grep|awk, using awk simply to print a column. After taking some time to actually read the gawk user manual a few years ago, I've found that I can do most things in awk that I was previously using grep/sed (and to a lesser extent) perl/python for in the shell. In my environment, it also comes with the benefit that I know the awk code I'm writing is supported by the version of gawk that's installed on every server/computer I come into contact with -- no having to ensure the right python/perl is available. One thing I wish there was better documentation on, though, is creating complete gawk scripts. With @include directives coupled with AWK_PATH, it's really convenient to build up a handful of utility functions that do frequent things and I've found, on several occasions, that I end up writing an .awk script instead of a bash/zsh script and it ends up being a much more straight-forward set of code. [0] Well, specifically, the 'gawk' variant

ausjke 8y ago
why not write something about the @include directives like a blog or some posts, that will be awesome.

wernsey 8y ago

One thing I also really like about Awk is that you can keep 99% of the language in your head. With Awk it is seldom necessary to even reference the man page. I don't have to interrupt myself with looking up the syntax or the order of the parameters to a function.

gitgacutils 8y ago

awk is a fundamental part of the unix ecosystem. It's in the hall of fame of utils along with the likes of grep. If you use unix or linux in any semi-serious capacity, you are familiar with awk.