8 ms·

"""Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same fu

"""Typical code density ratio of Python versus AWK is 10:1, sometimes more. This means that if you have a 650 line Python program, you can implement the same functionality in about 280 lines of AWK, and the program will be far simpler. I've once collapsed a 280+ line Python program into a simple 15 lines of code in AWK.""" How does this work? I am not saying it can't be done, but the main benefit of Awk seems to be quick one-liners, which are possible because you get "records" (splitting on whitespace) and lines (splitting on newline) and looping for free. But for larger programs, this easily translates to Python; just call readlines(), loop over it, call split() on each line. I would think that at this point, Awk doesn't have much of an advantage anymore... but apparently your experiences are different. What are some Awk constructs that would take a lot more code in Python?

empthought 9y ago
Every pattern being matched to every line can be a big win in more complex processing. This is a simple but familiar example: seq 1 30 | awk ' $0 % 3 == 0 { printf("Fizz"); replaced = 1 } $0 % 5 == 0 { printf("Buzz"); replaced = 1 } replaced { replaced = 0; printf("\n"); next } { print }' Note that the awk script is far more general than the typical interview question, which specifies the numbers to be iterated in order. The awk script works on any sequence of numbers.
- microtherion 9y ago
  Yes, but as zephyrfalcon said, that maps onto a series of if statements in python. No 10:1 magic anywhere.
  - empthought 9y ago
    The "series of if statements" also has to read the line, split it, and parse an integer. To behave like the AWK script it also has to catch an exception and continue when the input cannot be parsed as an integer. Go ahead, write the Python script that behaves exactly as this AWK program does. It will likely be 4x as long, and that's because the number of different patterns and actions to take is quite low. More complex (and hence more situated and less easy-to-understand) use cases will benefit even more from AWK's defaults. Moreover the pattern expressions are not constrained to simple tests: https://www.gnu.org/software/gawk/manual/html_node/Pattern-Overview.html#Pattern-Overview https://www.gnu.org/software/gawk/manual/html_node/Pattern-O... They can match ranges, regular expressions, or indeed any AWK expression. They can use the variables managed by the AWK interpreter: https://www.gnu.org/software/gawk/manual/html_node/Auto_002dset.html#Auto_002dset https://www.gnu.org/software/gawk/manual/html_node/Auto_002d... (NR and NF are commonly used). Actions one-way or two-way communicate with coprocesses with minimal ceremony: https://www.gnu.org/software/gawk/manual/html_node/Two_002dway-I_002fO.html#Two_002dway-I_002fO https://www.gnu.org/software/gawk/manual/html_node/Two_002dw... All of those mechanisms can be done in a Python script, but they add up to a lot of boilerplate and mindless yet error-prone translation to the standard library or Python looping and conditional logic.
    - microtherion 9y ago
      > The "series of if statements" also has to read the line, split it, and parse an integer All of which are built in functions... > To behave like the AWK script it also has to catch an exception and continue when the input cannot be parsed as an integer. Not quite sure what behavior you're referring to here. When I tested your script, it happily treated "xy" as divisible by 15. > Go ahead, write the Python script that behaves exactly as this AWK program does. import fileinput for line in fileinput.input(): replaced = False if int(line) % 3 == 0: print("Fizz", end=''); replaced = True if int(line) % 5 == 0: print("Buzz", end=''); replaced = True if replaced: print(); else: print(line, end='') > Moreover the pattern expressions are not constrained to simple tests And none of these, except maybe for the range operator, are particularly challenging for python.
      - empthought 9y ago
        1. Your script crashes when it is given input that does not parse as an integer. The awk script does not. In this way, the awk design favors robustness over correctness, which is a valid choice to make at times. 2. How would you modify it so it parsed a tab-delimited file and did FizzBuzz on the third column? With awk it is a simple matter of setting FS="\t" and changing $0 to $3? 3. How would you modify it so instead of being output unmodified, rows with $3 that are neither fizz nor buzz output the result of a subprocess called with the second column's contents? Now you might say that this is all goalpost-moving, but that's the point. AWK is more flexible and less cluttered in situations where the goalposts tend to get moved, but where the basic text processing paradigm stays the same.
        microtherion 9y ago
        1. Sure, it's a valid choice, and one that can easily be reproduced by python: def intish(str): try: return int(str) except: return 0 Can python's default be reproduced as easily in awk? 2. You'd insert field = line.split('\t') at the beginning of the loop and then refer to field[2] 3. os.popen or subprocess.run I buy the "less cluttered" argument when the problem matches awk's defaults. I vehemently disagree with the "more flexible" argument. A problem perfectly suited to awk can easily turn to a poor fit with the addition of a single, seemingly innocuous requirement (e.g. in your subprocess example, log the standard error of your subprocess into a separate file).
Annatar 9y ago
Hashed arrays are much simpler to do in AWK than they are in Python, for example. Also record splitting and processing is highly configurable in AWK with RS, ORS, OFS and one gets it for free without having to write extra code. And don’t forget that Python needs about 25,000 files just to fire up, while AWK is a single 169 KB executable (on Solaris / illumos / SmartOS). Makes a huge difference come application deployment time.

empthought 9y ago

Every pattern being matched to every line can be a big win in more complex processing. This is a simple but familiar example: seq 1 30 | awk ' $0 % 3 == 0 { printf("Fizz"); replaced = 1 } $0 % 5 == 0 { printf("Buzz"); replaced = 1 } replaced { replaced = 0; printf("\n"); next } { print }' Note that the awk script is far more general than the typical interview question, which specifies the numbers to be iterated in order. The awk script works on any sequence of numbers.

microtherion 9y ago
Yes, but as zephyrfalcon said, that maps onto a series of if statements in python. No 10:1 magic anywhere.
- empthought 9y ago
  The "series of if statements" also has to read the line, split it, and parse an integer. To behave like the AWK script it also has to catch an exception and continue when the input cannot be parsed as an integer. Go ahead, write the Python script that behaves exactly as this AWK program does. It will likely be 4x as long, and that's because the number of different patterns and actions to take is quite low. More complex (and hence more situated and less easy-to-understand) use cases will benefit even more from AWK's defaults. Moreover the pattern expressions are not constrained to simple tests: https://www.gnu.org/software/gawk/manual/html_node/Pattern-Overview.html#Pattern-Overview https://www.gnu.org/software/gawk/manual/html_node/Pattern-O... They can match ranges, regular expressions, or indeed any AWK expression. They can use the variables managed by the AWK interpreter: https://www.gnu.org/software/gawk/manual/html_node/Auto_002dset.html#Auto_002dset https://www.gnu.org/software/gawk/manual/html_node/Auto_002d... (NR and NF are commonly used). Actions one-way or two-way communicate with coprocesses with minimal ceremony: https://www.gnu.org/software/gawk/manual/html_node/Two_002dway-I_002fO.html#Two_002dway-I_002fO https://www.gnu.org/software/gawk/manual/html_node/Two_002dw... All of those mechanisms can be done in a Python script, but they add up to a lot of boilerplate and mindless yet error-prone translation to the standard library or Python looping and conditional logic.
  - microtherion 9y ago
    > The "series of if statements" also has to read the line, split it, and parse an integer All of which are built in functions... > To behave like the AWK script it also has to catch an exception and continue when the input cannot be parsed as an integer. Not quite sure what behavior you're referring to here. When I tested your script, it happily treated "xy" as divisible by 15. > Go ahead, write the Python script that behaves exactly as this AWK program does. import fileinput for line in fileinput.input(): replaced = False if int(line) % 3 == 0: print("Fizz", end=''); replaced = True if int(line) % 5 == 0: print("Buzz", end=''); replaced = True if replaced: print(); else: print(line, end='') > Moreover the pattern expressions are not constrained to simple tests And none of these, except maybe for the range operator, are particularly challenging for python.
    - empthought 9y ago
      1. Your script crashes when it is given input that does not parse as an integer. The awk script does not. In this way, the awk design favors robustness over correctness, which is a valid choice to make at times. 2. How would you modify it so it parsed a tab-delimited file and did FizzBuzz on the third column? With awk it is a simple matter of setting FS="\t" and changing $0 to $3? 3. How would you modify it so instead of being output unmodified, rows with $3 that are neither fizz nor buzz output the result of a subprocess called with the second column's contents? Now you might say that this is all goalpost-moving, but that's the point. AWK is more flexible and less cluttered in situations where the goalposts tend to get moved, but where the basic text processing paradigm stays the same.
      - microtherion 9y ago
        1. Sure, it's a valid choice, and one that can easily be reproduced by python: def intish(str): try: return int(str) except: return 0 Can python's default be reproduced as easily in awk? 2. You'd insert field = line.split('\t') at the beginning of the loop and then refer to field[2] 3. os.popen or subprocess.run I buy the "less cluttered" argument when the problem matches awk's defaults. I vehemently disagree with the "more flexible" argument. A problem perfectly suited to awk can easily turn to a poor fit with the addition of a single, seemingly innocuous requirement (e.g. in your subprocess example, log the standard error of your subprocess into a separate file).

Annatar 9y ago

Hashed arrays are much simpler to do in AWK than they are in Python, for example. Also record splitting and processing is highly configurable in AWK with RS, ORS, OFS and one gets it for free without having to write extra code. And don’t forget that Python needs about 25,000 files just to fire up, while AWK is a single 169 KB executable (on Solaris / illumos / SmartOS). Makes a huge difference come application deployment time.