8 ms·

I'm always happy when I see posts that promote AWK. It's a very underappreciated tool in my opinion. I was a Linux user for 20 years before I got familiar with

I'm always happy when I see posts that promote AWK. It's a very underappreciated tool in my opinion. I was a Linux user for 20 years before I got familiar with it. AWK is super powerful for text processing, and I like that it's included in Busybox for use on the embedded systems that I design. For any complex text processing, it's way better and more robust than having a super long pipeline of a bunch of sed/grep. Most recently, I used awk in a script that parses /proc/mount to grab the mountpoint of a partition, or print something different if the partition isn't mounted. Doable with a bunch of sed/grep and some shell logic? Definitely. But easier and cleaner in AWK, and equally easy to inline in a shell-script.

invisible 5y ago
For some purposes, awk+xargs can replace hours of work to write a tool to automate some process. It's my go-to for ops work that I don't expect to live very long and just needs to _happen_. Also, happy 1337 karma day :).
- 5e92cb50239222b 5y ago
  > awk+xargs can replace hours of work Including machine hours of work. Wasn't there a famous story of replacing a Hadoop cluster with an awk script (which was a couple orders of magnitude faster)? Oh yes, there was: https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841
  - dapids 5y ago
    In fairness it's xargs that is providing the command parallelization, not awk, but I agree both combined are a good match.
  - genewitch 5y ago
    If one considers the idea of map reduce to be taking a set of data and ending up with a subset that is relevant, I've used tons of simple things to do that, and never Hadoop. I think parsing logs to find pain areas or potential exploit/exfil is a map reduce job, for instance, and grep or awk can manage that just fine.
kbenson 5y ago
I've never bothered to learn much AWK, but that's mostly because Perl is my bread and butter language and has been for 20 years, and focusing on knowledge of that seemed a better investment (especially since with a few judicious flags, Perl is a passable AWK replacement even for very small one liners). That said, if you just want to supplement your knowledge of other shell tools and pull out something that can do some obvious text munging, AWK has always looked attractive for the task to me.
- chasil 5y ago
  The problem is that awk is in POSIX, and perl is not. There are two common sources of awk for Windows, for example, that drop one exe to provide the interpreter: http://unxutils.sourceforge.net/ http://unxutils.sourceforge.net/ https://frippery.org/busybox/ https://frippery.org/busybox/ Perl simply wasn't designed to do that.
  - newaccount2021 5y ago
    But perl is available by default in almost every free *nix, and for most people, Windows isn't a requirement
    - sigzero 5y ago
      Usually not a requirement but it's a "nice to have" sometimes.
    - chasil 5y ago
      In HP-UX 10.20, there was a single perl4 binary that dates from 1993 in /usr/contrib/bin (I checked our K380). Had Perl held to a single binary, it could be inside busybox. Perl cannot be inside busybox, and that hasn't been possible since 1993.
  - sigzero 5y ago
    https://sourceforge.net/projects/ezwinports/files/ https://sourceforge.net/projects/ezwinports/files/ Has a recent version of GAWK (Gnu AWK). I think if you install git for Windows, you get AWK as well.
SavantIdiot 5y ago
Came here to say this. Glad to see /bin getting respect. To anyone processing huge quantities of text and text files, someone very likely had the same problem you faced back in the 1980's and there's a Unix/GNU tool for it already.
- dylan604 5y ago
  I was introduced to *nix from processing very large text files that text editors I was familiar with choked and died. Someone showed me sed/awk/grep, and it took seconds to process when other GUI editors couldn't open the file. Never looked back.
GekkePrutser 5y ago
Not having to parse the output at all is even better. I really like the way Powershell can pass structured data like this. I'm a huge Linux/Unix fan but sometimes a rethink really works out. I hope Linux will get something similar. I know Powershell is available for Linux but without an adapted userland there's not much benefit
throwaway894345 5y ago
I do a lot of work with structured data--json, yaml, etc. For me, this is how I feel about jq. One of my favorite use-cases is querying Kubernetes resources. E.g., `kubectl get secret <secret-name> -o json | jq -r '.data | map_values(@base64d)'` (fetch a secret and decode all of its values).
Lio 5y ago
Yep awk is lovely and well worth the time to learn. This is probably not important for embedded but doesn’t a pipeline of small scripts (which could be in awk) give you better threading support? Xargs, GNU parallel or even make then scale that out really quickly.
zadwang 5y ago
I have used it for many thing such as extract bibtex entries and checking for missed definition of abbreviations, converting remind entry to ICS format etc. Recent example is to print a calendar like cal but with calendar dates highlighted.

invisible 5y ago

For some purposes, awk+xargs can replace hours of work to write a tool to automate some process. It's my go-to for ops work that I don't expect to live very long and just needs to _happen_. Also, happy 1337 karma day :).

5e92cb50239222b 5y ago
> awk+xargs can replace hours of work Including machine hours of work. Wasn't there a famous story of replacing a Hadoop cluster with an awk script (which was a couple orders of magnitude faster)? Oh yes, there was: https://news.ycombinator.com/item?id=17135841 https://news.ycombinator.com/item?id=17135841
- dapids 5y ago
  In fairness it's xargs that is providing the command parallelization, not awk, but I agree both combined are a good match.
- genewitch 5y ago
  If one considers the idea of map reduce to be taking a set of data and ending up with a subset that is relevant, I've used tons of simple things to do that, and never Hadoop. I think parsing logs to find pain areas or potential exploit/exfil is a map reduce job, for instance, and grep or awk can manage that just fine.

kbenson 5y ago

I've never bothered to learn much AWK, but that's mostly because Perl is my bread and butter language and has been for 20 years, and focusing on knowledge of that seemed a better investment (especially since with a few judicious flags, Perl is a passable AWK replacement even for very small one liners). That said, if you just want to supplement your knowledge of other shell tools and pull out something that can do some obvious text munging, AWK has always looked attractive for the task to me.

chasil 5y ago
The problem is that awk is in POSIX, and perl is not. There are two common sources of awk for Windows, for example, that drop one exe to provide the interpreter: http://unxutils.sourceforge.net/ http://unxutils.sourceforge.net/ https://frippery.org/busybox/ https://frippery.org/busybox/ Perl simply wasn't designed to do that.
- newaccount2021 5y ago
  But perl is available by default in almost every free *nix, and for most people, Windows isn't a requirement
  - sigzero 5y ago
    Usually not a requirement but it's a "nice to have" sometimes.
  - chasil 5y ago
    In HP-UX 10.20, there was a single perl4 binary that dates from 1993 in /usr/contrib/bin (I checked our K380). Had Perl held to a single binary, it could be inside busybox. Perl cannot be inside busybox, and that hasn't been possible since 1993.
- sigzero 5y ago
  https://sourceforge.net/projects/ezwinports/files/ https://sourceforge.net/projects/ezwinports/files/ Has a recent version of GAWK (Gnu AWK). I think if you install git for Windows, you get AWK as well.

SavantIdiot 5y ago

Came here to say this. Glad to see /bin getting respect. To anyone processing huge quantities of text and text files, someone very likely had the same problem you faced back in the 1980's and there's a Unix/GNU tool for it already.

dylan604 5y ago
I was introduced to *nix from processing very large text files that text editors I was familiar with choked and died. Someone showed me sed/awk/grep, and it took seconds to process when other GUI editors couldn't open the file. Never looked back.

GekkePrutser 5y ago

Not having to parse the output at all is even better. I really like the way Powershell can pass structured data like this. I'm a huge Linux/Unix fan but sometimes a rethink really works out. I hope Linux will get something similar. I know Powershell is available for Linux but without an adapted userland there's not much benefit

throwaway894345 5y ago

I do a lot of work with structured data--json, yaml, etc. For me, this is how I feel about jq. One of my favorite use-cases is querying Kubernetes resources. E.g., `kubectl get secret <secret-name> -o json | jq -r '.data | map_values(@base64d)'` (fetch a secret and decode all of its values).

Lio 5y ago

Yep awk is lovely and well worth the time to learn. This is probably not important for embedded but doesn’t a pipeline of small scripts (which could be in awk) give you better threading support? Xargs, GNU parallel or even make then scale that out really quickly.

zadwang 5y ago

I have used it for many thing such as extract bibtex entries and checking for missed definition of abbreviations, converting remind entry to ICS format etc. Recent example is to print a calendar like cal but with calendar dates highlighted.