10 ms·

Some people might think "why learn awk today when there is Python, Go, etc". But a lot of tasks (even big ones) can be done easier with this tool. The fact th

Some people might think "why learn awk today when there is Python, Go, etc". But a lot of tasks (even big ones) can be done easier with this tool. The fact that most OSs have a POSIX compliant version of it in the base system also make it very valuable, not just for the people crunching data but also to sysadmin/devops people.

digi_owl 8y ago
And are probably less of a mess to get installed and running...
- Spivak 8y ago
  Yeah, you can never be sure what version of a scripting language is going to be installed on a system. Always a little risky depending on the system version. These tools have the advantage of having not changed too much in the last few decades.
jandrese 8y ago
Awk works great in pipes on the commandline, unlike Python where you would be writing a multiline script to accomplish the same task. My 99% use case is extracting particular fields from some input stream. awk '{print $1,$5}' That's a tremendously common problem and awk is the best tool for the job. I once started diving deeper into awk and there is an impressive amount you can do with it, but I hit a point pretty quickly where it would be cleaner to write the script in a more traditional language where my successor won't be cursing my name for writing something complex in an obscure language.
- Spivak 8y ago
  If you just want to extract fields the cut utility might be a better fit. cut -d $delim -f $field1,$field2,...
  - mmt 8y ago
    I'll tend to do this in scripts, both for clarity and precision (i.e. when I know the specific, single delimiter). However, for ad-hoc command-line usage, I still prefer awk, because (1) the default delimiter is white space, which means I don't have to worry about if what I'm looking at might contain tabs intermingled with spaces (or how many spaces) and (2) it makes it much easier to incrementally increase the complexity of my ad-hoc operation, such as if I decide I want to take a sum or average of one of those columns, after all.
  - kjeetgill 8y ago
    I first learned awk precisely to replace cut. Not being able to reorder fields make it an especially narrow tool. Is there anything from cut I'm missing out on?
    - mmt 8y ago
      > Is there anything from cut I'm missing out on? It's a narrow tool. Other than the inherent advantage this brings in clarity/simplicity in a script, there's nothing else to it.
      - kazinator 8y ago
        Scripts in which you have both cut and awk are not simpler or clearer than ones in which you only use awk.
        mmt 8y ago
        Although I completely agree with this for any given "one-liner" or pipeline (even if it's multiple lines), I don't believe the mere use of awk in one place in the script means using cut elsewhere isn't simpler/clearer there. Consider a script that originally had now awk but just a field=$(somecmd | cut -d: -f3) but now has been modified to have awk elsewhere. Did the above just become less simple or clear? If so, is it worth changing it to the below for clarity? Does it matter if the awk predates the cut, instead? field=$(somecmd | awk -F: '{print $2}') I say "no" to both, although I do recognize the argument for just using the same, consistent tool everywhere, instead.
- kakwa_ 8y ago
  It's also use awk in that manner 99% of the time. Another common usage for awk in my case is to do floating operations (*, +, eq, gt, etc) in a shell script. It avoids installing bc, which is not always present by default. I used it for other things, like for example comparing software versions in shell scripts. The most advanced usage of awk in my case was to do some statistic analysis (average, confidence interval, standard deviation) of network simulations.
  - lozf 8y ago
    Just be aware there are some caveats when using awk to operate on large numbers. GNU awk has the -M option to handle precision arithmetic, but where you can't use `gawk`, `bc` may be safer afterall.
- hawski 8y ago
  I like awk very much, but cut has one big advantage over awk - you can easily output range of fields also open ended ones. Example: $ cat foo foo bar baz foo qux $ < foo cut -d' ' -f 2- bar baz qux As far as I know you have to loop through the fields, which is (haha) awkward.
kakwa_ 8y ago
> The fact that most OSs have a POSIX compliant version of it You need to be cautious however, gnu awk can be somewhat inconsistent at times: try the following on various distros (and also an *BSD) for example: awk 'BEGIN {print "9.1" >= "8"}'; awk 'BEGIN {print "9.1" > "8"}'; awk 'BEGIN {print "9.1" < "8"}'; awk 'BEGIN {print "9.1" <= "8"}'; on OpenBDS, you get no output or a syntax error, on Debian, you get the correct result for < and <= in all cases, never for > (it's interpreted as a redirection), and >= only works with newer Debian versions (sid and maybe stretch). I know that the fix is simple: just put some parenthesis (awk 'BEGIN {print ("9.1" <= "8")}'), but I was quite amused to discover that a few days ago.
dredmorbius 8y ago
Most busybox installations include awk. This is a literal lifesaver.
kumarharsh 8y ago
I agree! I absolutely love Javascript, but in the time it'll take me to setup a project, decide whether to use JS/Typescript/Flow and to actually write the code I would have done the goddamn thing in awk, gone home and watched Ronaldo score the hat-trick.
oldcynic 8y ago
Most of my early career was Unix, with just a little DOS and Windows thrown in (enough to persuade me to keep trying to avoid it). Unix was mature even then. There is NO WAY you would have convinced me that 30 years later I'd still be using most of that, still gluing things together with the same shell tools. Or that most of the books would still have a place on the shelf (albeit sometimes in 2nd or 3rd editions), and turn out to be classics. Ignoring big iron, we'd gone from Apple IIs to Amigas and workstations in just a few years, and hardware was changing as rapidly as the current fashion for JS frameworks does today. Yet here we are, awk, grep, sed and the other tools still feel like the right, concise answer. So why not learn them, they might greatly surprise you by their staying power.
rspeer 8y ago
Also, awk is way faster than Python. Its patterns aren't interpreted over and over like they would be in Python; they're compiled into a state machine.
- Too 8y ago
  Python regexes can be saved into variables, and even if you don't do it manually it automatically maintains a pattern-cache for the last expressions you have used.
  - rspeer 8y ago
    Maybe I wasn't clear about this distinction. Python is an interpreted language. Even if you are matching a "compiled" regex, you are repeatedly interpreting the line that says to match it. Awk is a compiled language. Your Awk script is compiled once and applied to every line of your file at C-like speeds. It is way faster than Python. If you learn to use Awk well, you will start doing things with data that you wouldn't have had the patience to do in an interpreted language.
    - sametmax 8y ago
      Technically, python is compiled to pyc and not reparsed again and again. What'd more, string operationd are all implemented in c. Unless you use pypy, which is faster.
      - rspeer 8y ago
        Trust me. Awk is much faster. I say this with no disrespect for Python, it's just you should know the tradeoffs of the tools you use. It does not matter if the individual operations in Python are implemented in C. It will still be much slower at looping over lines of a file than a compiled language designed decades ago for that exact purpose.
        sametmax 8y ago
        I'm really unsure about my benchmarking capabilities, yet I wanted to have a general idea of the order of magnitude we are talking about. Reading your comment, I was expecting at least a X5 speed up by using awk. I have a lot of Python file in one dir: $ find . -iname "*.py" | wc -l 10429 Finding them and cating them all takes about 0.3 secs: $ time find . -iname "*.py" | xargs cat {} > /tmp/cat.out ... real 0m0.344s user 0m0.140s sys 0m0.175s So to have something simple that takes a bit of time, I tried to get all lines starting with "print", and output the first thing after that. I'm really bad at awk, so I don't know if there is a better way. I went for the most obvious thing for me: $ time find . -iname "*.py" | xargs cat {} | awk '/^print/{print($2)}' > /tmp/awk.out ... real 0m1.111s user 0m1.165s sys 0m0.368s Now, with Python, it's definitely not as easy to type. You have to get a script like. awk wins the expressivity metrics for this use case: import sys for x in sys.stdin: if x.startswith('print'): try: print(x.split()[1]) except IndexError: print('') # to match awk behavior But as for performance, I don't get the huge boost in perfs you are talking about: $ time find . -iname "*.py" | xargs cat {} | python /tmp/test.py > /tmp/python.out ... real 0m0.762s user 0m0.862s sys 0m0.347s I do get the same output though: $ cmp /tmp/python.out /tmp/awk.out && echo "yes" yes
busterarm 8y ago
You wouldn't believe how many complicated python tasks I see that run into the many hundreds of lines of code that are an awk one or two liner and a line in crontab. And if you really want that in a repo, you can stick that in ansible trivially. Sometimes instead I just wrap a line of awk in %x() in Ruby to save a few chunks of code. Most programmers don't understand why I do this though. :(
- sametmax 8y ago
  > You wouldn't believe how many complicated python tasks I see that run into the many hundreds of lines of code that are an awk one or two liner and a line in crontab. Hundred of lines of python by two awk lines ? Must be very bad python programmers.

digi_owl 8y ago

And are probably less of a mess to get installed and running...

Spivak 8y ago
Yeah, you can never be sure what version of a scripting language is going to be installed on a system. Always a little risky depending on the system version. These tools have the advantage of having not changed too much in the last few decades.

jandrese 8y ago

Awk works great in pipes on the commandline, unlike Python where you would be writing a multiline script to accomplish the same task. My 99% use case is extracting particular fields from some input stream. awk '{print $1,$5}' That's a tremendously common problem and awk is the best tool for the job. I once started diving deeper into awk and there is an impressive amount you can do with it, but I hit a point pretty quickly where it would be cleaner to write the script in a more traditional language where my successor won't be cursing my name for writing something complex in an obscure language.

Spivak 8y ago
If you just want to extract fields the cut utility might be a better fit. cut -d $delim -f $field1,$field2,...
- mmt 8y ago
  I'll tend to do this in scripts, both for clarity and precision (i.e. when I know the specific, single delimiter). However, for ad-hoc command-line usage, I still prefer awk, because (1) the default delimiter is white space, which means I don't have to worry about if what I'm looking at might contain tabs intermingled with spaces (or how many spaces) and (2) it makes it much easier to incrementally increase the complexity of my ad-hoc operation, such as if I decide I want to take a sum or average of one of those columns, after all.
- kjeetgill 8y ago
  I first learned awk precisely to replace cut. Not being able to reorder fields make it an especially narrow tool. Is there anything from cut I'm missing out on?
  - mmt 8y ago
    > Is there anything from cut I'm missing out on? It's a narrow tool. Other than the inherent advantage this brings in clarity/simplicity in a script, there's nothing else to it.
    - kazinator 8y ago
      Scripts in which you have both cut and awk are not simpler or clearer than ones in which you only use awk.
      - mmt 8y ago
        Although I completely agree with this for any given "one-liner" or pipeline (even if it's multiple lines), I don't believe the mere use of awk in one place in the script means using cut elsewhere isn't simpler/clearer there. Consider a script that originally had now awk but just a field=$(somecmd | cut -d: -f3) but now has been modified to have awk elsewhere. Did the above just become less simple or clear? If so, is it worth changing it to the below for clarity? Does it matter if the awk predates the cut, instead? field=$(somecmd | awk -F: '{print $2}') I say "no" to both, although I do recognize the argument for just using the same, consistent tool everywhere, instead.
kakwa_ 8y ago
It's also use awk in that manner 99% of the time. Another common usage for awk in my case is to do floating operations (*, +, eq, gt, etc) in a shell script. It avoids installing bc, which is not always present by default. I used it for other things, like for example comparing software versions in shell scripts. The most advanced usage of awk in my case was to do some statistic analysis (average, confidence interval, standard deviation) of network simulations.
- lozf 8y ago
  Just be aware there are some caveats when using awk to operate on large numbers. GNU awk has the -M option to handle precision arithmetic, but where you can't use `gawk`, `bc` may be safer afterall.
hawski 8y ago
I like awk very much, but cut has one big advantage over awk - you can easily output range of fields also open ended ones. Example: $ cat foo foo bar baz foo qux $ < foo cut -d' ' -f 2- bar baz qux As far as I know you have to loop through the fields, which is (haha) awkward.

kakwa_ 8y ago

> The fact that most OSs have a POSIX compliant version of it You need to be cautious however, gnu awk can be somewhat inconsistent at times: try the following on various distros (and also an *BSD) for example: awk 'BEGIN {print "9.1" >= "8"}'; awk 'BEGIN {print "9.1" > "8"}'; awk 'BEGIN {print "9.1" < "8"}'; awk 'BEGIN {print "9.1" <= "8"}'; on OpenBDS, you get no output or a syntax error, on Debian, you get the correct result for < and <= in all cases, never for > (it's interpreted as a redirection), and >= only works with newer Debian versions (sid and maybe stretch). I know that the fix is simple: just put some parenthesis (awk 'BEGIN {print ("9.1" <= "8")}'), but I was quite amused to discover that a few days ago.

dredmorbius 8y ago

Most busybox installations include awk. This is a literal lifesaver.

kumarharsh 8y ago

I agree! I absolutely love Javascript, but in the time it'll take me to setup a project, decide whether to use JS/Typescript/Flow and to actually write the code I would have done the goddamn thing in awk, gone home and watched Ronaldo score the hat-trick.

oldcynic 8y ago

Most of my early career was Unix, with just a little DOS and Windows thrown in (enough to persuade me to keep trying to avoid it). Unix was mature even then. There is NO WAY you would have convinced me that 30 years later I'd still be using most of that, still gluing things together with the same shell tools. Or that most of the books would still have a place on the shelf (albeit sometimes in 2nd or 3rd editions), and turn out to be classics. Ignoring big iron, we'd gone from Apple IIs to Amigas and workstations in just a few years, and hardware was changing as rapidly as the current fashion for JS frameworks does today. Yet here we are, awk, grep, sed and the other tools still feel like the right, concise answer. So why not learn them, they might greatly surprise you by their staying power.

rspeer 8y ago

Also, awk is way faster than Python. Its patterns aren't interpreted over and over like they would be in Python; they're compiled into a state machine.

Too 8y ago
Python regexes can be saved into variables, and even if you don't do it manually it automatically maintains a pattern-cache for the last expressions you have used.
- rspeer 8y ago
  Maybe I wasn't clear about this distinction. Python is an interpreted language. Even if you are matching a "compiled" regex, you are repeatedly interpreting the line that says to match it. Awk is a compiled language. Your Awk script is compiled once and applied to every line of your file at C-like speeds. It is way faster than Python. If you learn to use Awk well, you will start doing things with data that you wouldn't have had the patience to do in an interpreted language.
  - sametmax 8y ago
    Technically, python is compiled to pyc and not reparsed again and again. What'd more, string operationd are all implemented in c. Unless you use pypy, which is faster.
    - rspeer 8y ago
      Trust me. Awk is much faster. I say this with no disrespect for Python, it's just you should know the tradeoffs of the tools you use. It does not matter if the individual operations in Python are implemented in C. It will still be much slower at looping over lines of a file than a compiled language designed decades ago for that exact purpose.
      - sametmax 8y ago
        I'm really unsure about my benchmarking capabilities, yet I wanted to have a general idea of the order of magnitude we are talking about. Reading your comment, I was expecting at least a X5 speed up by using awk. I have a lot of Python file in one dir: $ find . -iname "*.py" | wc -l 10429 Finding them and cating them all takes about 0.3 secs: $ time find . -iname "*.py" | xargs cat {} > /tmp/cat.out ... real 0m0.344s user 0m0.140s sys 0m0.175s So to have something simple that takes a bit of time, I tried to get all lines starting with "print", and output the first thing after that. I'm really bad at awk, so I don't know if there is a better way. I went for the most obvious thing for me: $ time find . -iname "*.py" | xargs cat {} | awk '/^print/{print($2)}' > /tmp/awk.out ... real 0m1.111s user 0m1.165s sys 0m0.368s Now, with Python, it's definitely not as easy to type. You have to get a script like. awk wins the expressivity metrics for this use case: import sys for x in sys.stdin: if x.startswith('print'): try: print(x.split()[1]) except IndexError: print('') # to match awk behavior But as for performance, I don't get the huge boost in perfs you are talking about: $ time find . -iname "*.py" | xargs cat {} | python /tmp/test.py > /tmp/python.out ... real 0m0.762s user 0m0.862s sys 0m0.347s I do get the same output though: $ cmp /tmp/python.out /tmp/awk.out && echo "yes" yes

busterarm 8y ago

You wouldn't believe how many complicated python tasks I see that run into the many hundreds of lines of code that are an awk one or two liner and a line in crontab. And if you really want that in a repo, you can stick that in ansible trivially. Sometimes instead I just wrap a line of awk in %x() in Ruby to save a few chunks of code. Most programmers don't understand why I do this though. :(

sametmax 8y ago
> You wouldn't believe how many complicated python tasks I see that run into the many hundreds of lines of code that are an awk one or two liner and a line in crontab. Hundred of lines of python by two awk lines ? Must be very bad python programmers.