4 ms·

AWK is a programming language. In the AWK book by Aho, Weinberger and Kernighan, towards the end of the book they implement an arbitrary assembler, the virtual

AWK is a programming language. In the AWK book by Aho, Weinberger and Kernighan, towards the end of the book they implement an arbitrary assembler, the virtual processor and a virtual machine for the machine code they just invented to run. They also implement a relational database management system in AWK, as well as an autoscale graphing solution. I myself have implemented XML SOAP command line client, a backup solution, a SAN UUID management application and an automated Oracle RAC SAN storage migration solution, a configuration management, and an Oracle database creation / management applications in AWK. Usually I develop a thin getopts shell wrapper around an AWK core. Works every time, the executables are on the order of a few KB (the largest so far, the XML SOAP client is 24.5 KB) and they all run like a bandit. Memory requirements are miniscule. Dependencies are minimal: the only external dependency so far in my software has been the xsltproc binary from the libxslt package. AWK is easier to use than Python or Perl, and is much faster than either of those. Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same functionality in about 280 lines of AWK, and the program will be far simpler. I've once collapsed a 280+ line Python program into a simple 15 lines of code in AWK. AWK is an extremely versatile, powerful programming language. For even more speed, AWKA can be used to transpile AWK source into C and then it will call an optimizing C compiler to compile it into a binary executable. Typical speedup is on the order of 100%, so if your AWK program ran in 12 seconds, it'll now finish in six.

segmondy 9y ago
I once introduced AWK to a team and shared the same book with them. They didn't know that AWK was a programming language. Told them they could achieve more faster with AWK than using python & php for transforming and shuffling around data. They looked at me like I was crazy. :(
zephyrfalcon 9y ago
"""Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same functionality in about 280 lines of AWK, and the program will be far simpler. I've once collapsed a 280+ line Python program into a simple 15 lines of code in AWK.""" How does this work? I am not saying it can't be done, but the main benefit of Awk seems to be quick one-liners, which are possible because you get "records" (splitting on whitespace) and lines (splitting on newline) and looping for free. But for larger programs, this easily translates to Python; just call readlines(), loop over it, call split() on each line. I would think that at this point, Awk doesn't have much of an advantage anymore... but apparently your experiences are different. What are some Awk constructs that would take a lot more code in Python?
- empthought 9y ago
  Every pattern being matched to every line can be a big win in more complex processing. This is a simple but familiar example: seq 1 30 | awk ' $0 % 3 == 0 { printf("Fizz"); replaced = 1 } $0 % 5 == 0 { printf("Buzz"); replaced = 1 } replaced { replaced = 0; printf("\n"); next } { print }' Note that the awk script is far more general than the typical interview question, which specifies the numbers to be iterated in order. The awk script works on any sequence of numbers.
  - microtherion 9y ago
    Yes, but as zephyrfalcon said, that maps onto a series of if statements in python. No 10:1 magic anywhere.
    - empthought 9y ago
      The "series of if statements" also has to read the line, split it, and parse an integer. To behave like the AWK script it also has to catch an exception and continue when the input cannot be parsed as an integer. Go ahead, write the Python script that behaves exactly as this AWK program does. It will likely be 4x as long, and that's because the number of different patterns and actions to take is quite low. More complex (and hence more situated and less easy-to-understand) use cases will benefit even more from AWK's defaults. Moreover the pattern expressions are not constrained to simple tests: https://www.gnu.org/software/gawk/manual/html_node/Pattern-Overview.html#Pattern-Overview https://www.gnu.org/software/gawk/manual/html_node/Pattern-O... They can match ranges, regular expressions, or indeed any AWK expression. They can use the variables managed by the AWK interpreter: https://www.gnu.org/software/gawk/manual/html_node/Auto_002dset.html#Auto_002dset https://www.gnu.org/software/gawk/manual/html_node/Auto_002d... (NR and NF are commonly used). Actions one-way or two-way communicate with coprocesses with minimal ceremony: https://www.gnu.org/software/gawk/manual/html_node/Two_002dway-I_002fO.html#Two_002dway-I_002fO https://www.gnu.org/software/gawk/manual/html_node/Two_002dw... All of those mechanisms can be done in a Python script, but they add up to a lot of boilerplate and mindless yet error-prone translation to the standard library or Python looping and conditional logic.
microtherion 9y ago
I have quite fond memories of awk, but some of these claims might be a bit on the [citation needed] side. "Easier to use" - maybe so, on the particular subset of problems that awk was designed for. However, the ease of use upside is limited - awk constructs map pretty much 1:1 onto Python/Perl constructs that are not particularly complicated. Conversely, there is a vast set of problems that are still straightforward to solve in Python/Perl and would be rather awkward in awk. "much faster" - the comparisons I've seen (and done) usually had awk and perl5 roughly at parity. "code density ratio 10:1" - I call BS on that one. Sure, with the benefit of hindsight, it's sometimes possible to vastly simplify a script, but that has little to do with the languages involved. There is no awk solution that cannot be expressed in about 2x the lines of Python code (and that 2x is mostly because idiomatic awk puts conditions and code on one line, while Python puts them on two lines).

segmondy 9y ago

I once introduced AWK to a team and shared the same book with them. They didn't know that AWK was a programming language. Told them they could achieve more faster with AWK than using python & php for transforming and shuffling around data. They looked at me like I was crazy. :(

zephyrfalcon 9y ago

"""Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same functionality in about 280 lines of AWK, and the program will be far simpler. I've once collapsed a 280+ line Python program into a simple 15 lines of code in AWK.""" How does this work? I am not saying it can't be done, but the main benefit of Awk seems to be quick one-liners, which are possible because you get "records" (splitting on whitespace) and lines (splitting on newline) and looping for free. But for larger programs, this easily translates to Python; just call readlines(), loop over it, call split() on each line. I would think that at this point, Awk doesn't have much of an advantage anymore... but apparently your experiences are different. What are some Awk constructs that would take a lot more code in Python?

empthought 9y ago
Every pattern being matched to every line can be a big win in more complex processing. This is a simple but familiar example: seq 1 30 | awk ' $0 % 3 == 0 { printf("Fizz"); replaced = 1 } $0 % 5 == 0 { printf("Buzz"); replaced = 1 } replaced { replaced = 0; printf("\n"); next } { print }' Note that the awk script is far more general than the typical interview question, which specifies the numbers to be iterated in order. The awk script works on any sequence of numbers.
- microtherion 9y ago
  Yes, but as zephyrfalcon said, that maps onto a series of if statements in python. No 10:1 magic anywhere.
  - empthought 9y ago
    The "series of if statements" also has to read the line, split it, and parse an integer. To behave like the AWK script it also has to catch an exception and continue when the input cannot be parsed as an integer. Go ahead, write the Python script that behaves exactly as this AWK program does. It will likely be 4x as long, and that's because the number of different patterns and actions to take is quite low. More complex (and hence more situated and less easy-to-understand) use cases will benefit even more from AWK's defaults. Moreover the pattern expressions are not constrained to simple tests: https://www.gnu.org/software/gawk/manual/html_node/Pattern-Overview.html#Pattern-Overview https://www.gnu.org/software/gawk/manual/html_node/Pattern-O... They can match ranges, regular expressions, or indeed any AWK expression. They can use the variables managed by the AWK interpreter: https://www.gnu.org/software/gawk/manual/html_node/Auto_002dset.html#Auto_002dset https://www.gnu.org/software/gawk/manual/html_node/Auto_002d... (NR and NF are commonly used). Actions one-way or two-way communicate with coprocesses with minimal ceremony: https://www.gnu.org/software/gawk/manual/html_node/Two_002dway-I_002fO.html#Two_002dway-I_002fO https://www.gnu.org/software/gawk/manual/html_node/Two_002dw... All of those mechanisms can be done in a Python script, but they add up to a lot of boilerplate and mindless yet error-prone translation to the standard library or Python looping and conditional logic.

microtherion 9y ago

I have quite fond memories of awk, but some of these claims might be a bit on the [citation needed] side. "Easier to use" - maybe so, on the particular subset of problems that awk was designed for. However, the ease of use upside is limited - awk constructs map pretty much 1:1 onto Python/Perl constructs that are not particularly complicated. Conversely, there is a vast set of problems that are still straightforward to solve in Python/Perl and would be rather awkward in awk. "much faster" - the comparisons I've seen (and done) usually had awk and perl5 roughly at parity. "code density ratio 10:1" - I call BS on that one. Sure, with the benefit of hindsight, it's sometimes possible to vastly simplify a script, but that has little to do with the languages involved. There is no awk solution that cannot be expressed in about 2x the lines of Python code (and that 2x is mostly because idiomatic awk puts conditions and code on one line, while Python puts them on two lines).