12 ms·

The Awk book’s 60-line version of Make

260 points by nalgeon 3y ago

version_five 3y ago
I'm a big awk fan but I'm not sold on this. The awk program is not very readable- I think that's fine for a dense one-liner, I'm not really sure it carries over to a 60 line script. I think for something like this I'd prefer a bash script, maybe with awk invoked somewhere, that would be much easier to understand at a glance. Is there something in the awk script that makes it advantageous over a shell script? Edit: I hadn't read the author's conclusion yet when I posted, he agrees I consider AWK amazing, but I think it should remain where it excels: for exploratory data analysis and for one-liner data extraction scripts
- Avshalom 3y ago
  Well I think generally a 60 line program fits in that spot of "write once, read never, start from scratch if it ever turns out to be inadequate" ... also known as the APL Zone
  - yjftsjthsd-h 3y ago
    I'm not dead set against it, but if there was any mistake or bugs I don't know how you'd find them and fix them in that approach
    - fuzztester 3y ago
      By checking the correctness of the outputs, which you need to do anyway?
      - edgyquant 3y ago
        The point is the “and fix them”
        deleted 3y ago
        [deleted]
      - macintux 3y ago
        Not only is fixing more difficult, but also looking for likely weaknesses (and thus the inputs and outputs to focus on for testing).
      - yjftsjthsd-h 3y ago
        Okay, so the first dev writes 60 lines of indecipherable code, runs some sample invocations, looks at the output, says it looks good. A few months later, someone - maybe the original dev, maybe some other sucker - notices that in some edge case the code misbehaves. Now what? (Obviously, any answer that involves "don't write code with bugs" or "write perfect tests" is a nonstarter)
        HALtheWise 3y ago
        If we're going with the "start from scratch if it ever proves inadequate" philosophy, then the person who notices the misbehavior looks at the original code, sees that it's written in some obscure language, is undecipherable, but also is only 60 lines long, and decides that it will probably be simpler to make a new (short) implementation in their own favorite language that correctly handles both the original use case and their new requirement. The key insight is that given how much easier it is to write fresh code than understand old stuff, they could very well be correct in that guess, and the end result is a single piece of small clean code, rather than a simple core with layers of patches glued on top. In this particular case, we're talking about a "make" replacement, so testing the new implementation can be done by simply running "make all" for the project. If it passes, then the new implementation must be identical to the old one in all the ways that actually matter for the project at hand. In all likelihood, for a simple program like this, fixing one bug will also silently fix others because the new architecture is probably better than the old one.
        falcor84 3y ago
        I actually really like this approach, and have been thinking about this in regards to coding with an LLM - for a sufficiently simple program (and assuming no security concerns), once you trust your test suite, you should trust AI generated code that passes it. And then if requirements change, you should be able to amend the test cases, rerun the AI until it passes all tests and linters, maybe give the code a quick glance, and be on with your life.
- Spivak 3y ago
  > for exploratory data analysis and for one-liner data extraction scripts I think both you and the author just don't like AWK if that's the takeaway. What you're describing is literally 1% of the AWK language -- like you don't have to like it, it's weird in many respects but you're treating AWK like it's jq when it's actually closer to like a Perl-Lite/Bash mix. An AWK focused on just those use-cases would look very different. One of my favorite resources on AWK: https://www.grymoire.com/Unix/Awk.html https://www.grymoire.com/Unix/Awk.html
- Someone 3y ago
  > The awk program is not very readable What do you find hard to read about it? If you know what make does, I think it is fairly easy to read, even for those who don’t know awk at all, but do know the Unix shell (to recognize ‘ls -t’) and C (both of which, probably the audience for this book knew, given that the book is from 1988) > I think for something like this I'd prefer a bash script But would it be easier to read? I doubt see why it would.
  - mauvehaus 3y ago
    Bash also would have been an unlikely choice for a book published in 1988, considering it wasn't released until 1989 (Per Wikipedia).
    - Brian_K_White 3y ago
      It would have been ksh, which was the bash of the day, as in, the more featureful sh-compatible sh-superset. But a bash or ksh script would have been less readable than awk. bash (or ksh88 or ksh93) is powrful and useful but not readable if you're actually using the powerful useful features. In bash, a lot of functionality comes in the form of brace expansions and word splitting, basically abusing the command parser to get results there is no actual function for. In awk and any other more normal programming language, those same features come in the form of an explicit function to do that thing.
      - fuzztester 3y ago
        >In bash, a lot of functionality comes in the form of brace expansions and word splitting, basically abusing the command parser to get results there is no actual function for. In awk and any other more normal programming language, those same features come in the form of an explicit function to do that thing. Right. That's one of the reasons why the man page for bash is so long. IIRC, going way back, even the page for plain sh was long, for the same reason.
        falcor84 3y ago
- mbivert 3y ago
  I think it should be appreciated in context: it's a good way to teach both awk(1) and make(1) to someone new to UNIX. It also demonstrates how to use awk(1) for prototyping, which IMO is a good programming habit to "develop": it forces to focus on the essential, and not to unnecessarily overthink.
- pushedx 3y ago
  Writing this make program in bash would invovle even more difficult to read hacks, as bash also does not support multidimensional arrays.
- Brian_K_White 3y ago
  A while ago I wrote a program to renumber TRS-80 Model 100 BASIC code in awk. Then re-wrote it in bash (pure bash, no sed/grep/cut/tr/awk/bc/etc), and the two are practically identical. That suprised me just how practically identical they wwere in the end. awk is like a hidden miracle of utility just sitting there unused on every machine since the dawn of time. Normally if you want something to be ultra portable, you write it in sh or ksh, (though by now, bash would be ok, I mean there is bash for xenix), but to get the most out of ksh or bash, you have to use all the available features and tricks that are powerful and useful but NOT readable. 50% of the logic of a given line of code is not spelled out in the keywords but in arcane brace expansion and word splitting rules. But every system that might have some version of bash or ksh or plain sh, always has awk too, and even the oldest plain not-gnu awk is a real, "normal", more or less straighforward explicit programming language compared to bash. Not all that much more pawerful, but more readable and more straightforward to write. Things are done with functions that take parameters and do things to the parameters, not with special syntax that does magic transformations of variables which you then parlay into various uses. Everyone uses perl/python/ruby/php/whatever when the project goes beyond bash scope, but they all need to be installed and need to be a particular version, and almost always need some library of modules as well, and every python script breaks every other year or on every other new platform. But awk is already there, even on ancient obscure systems that absolutely can not have the current version of python or ruby and all the gems. I don't use it for current day to day stuff either, there's too many common things today that it has no knowledge of. I don't want to try to do https transactions or parse xml in awk. I'm just saying it's interesting or somehow notable how generically useful awk is pretty much just like bash or python, installed everywhere already, and almost utterly unused.
- rottc0dd 3y ago
  Bash would really be bad idea if it is going to use bash stitching so many gnu utils for this kind of job. I once had to rewrite a bash script into awk[1] that is big enough and it made the program more readable and the total time execution came down from 12 mins to less than 1 second. I think maybe the original bash script would have written badly, (each util command will invoke it's own process and it has to piped to others instead of using awk which will be running in a single process). [1] - https://github.com/berry-thawson/diff2html/blob/master/diff2html.sh https://github.com/berry-thawson/diff2html/blob/master/diff2...
- camgunz 3y ago
  It seems pretty readable to me, in particular the "update" function parses as JavaScript if you fix the implicit string concatenation (template literals or +) and replace the # comments with //. I'm actually surprised JavaScript is so similar to awk; it feels like a descendant language tbh.
- tyingq 3y ago
  > Is there something in the awk script that makes it advantageous over a shell script? Pseudo multi-dimensional associative arrays for representing the dependency graph of make. This part: for (i = 2; i <= NF; i++) slist[nm, ++scnt[nm]] = $i The way awk supports them is hacky and not really a multidimensional array, but still is better than what you would have to do with bash, because of split() and some other language features. It would be much easier with any scripting language though, Perl for example.
- agateau 3y ago
  I would find it easier to read with more sensible, non-abreviated variable names.
1vuio0pswjnm7 3y ago
These blog posts and discussion usually pit one language against others and often attempt to restrict a language to some specific context, ignoring that each user's experience, needs and preferences may be different. A more interesting debate would be language-agnostic, such as writing one-liners versus writing lengthy programs. In short, the debate might be something like: What does the computer user prefer more: (a) writing one-liners or (b) writing lengthy programs. Not everyone will have the same answer. Knuth might prefer (b). McIllroy might prefer (a). Assuming one reading this blog post knew nothing about programming languages, it seems to imply Python is not well-suited for one-liners, or at least not comparable to AWK in that context. Perhaps the interpreter startup time might have something to do with the failure to consider Python for one-liners.
- benhoyt 3y ago
  I don't think Python is very well suited to one-liners, but it's not due to interpreter startup time (20ms on my machine). Rather, it's due to all the scaffolding needed, which AWK provides implicitly: AWK automatically reads input lines and splits them into fields, automatically initializes variables to the type's default value, and has terser syntax for things like regex matching. Consider the following AWK one-liner which, for every input line that starts with a letter, prints the line number and the line's second field: awk '/^[A-Za-z]/ { print NR, $2 }' The equivalent Python program has a ton more boilerplate: import statements, explicit input reading and field splitting, and more verbose regex matching: import re import fileinput inp = fileinput.input(encoding='utf-8') for line in inp: if re.match(r'[A-Za-z]', line): fields = line.split() print(inp.lineno(), fields[1])
  - fuzztester 3y ago
    >The equivalent Python program has a ton more boilerplate: import statements, explicit input reading and field splitting, and more verbose regex matching: "awks and pythons"
    - fuzztester 3y ago
kazinator 3y ago
Ben notes that Kernighan regrets the way local variables are handled in Awk. I patched GNU Awk to have a @let extension that gives you scoped locals (usable in functions as well as in BEGIN/END blocks): $ egawk 'BEGIN { x = 3; print x; @let (x = 4, y) { print x } print x }' 3 4 3 @ is used because there is at least one other existing extension which is like that: @include. https://www.kylheku.com/cgit/egawk/about/ https://www.kylheku.com/cgit/egawk/about/ This was rejected by the GNU Awk project, though. I was encouraged to make a fork and give it some kind of different name, so I did that.
- Brian_K_White 3y ago
  Did they cite a reason that sounded reasonable? Like the particular implementation breaks some design principle they want to stick to or something? Did they suggest it might be acceptable some other way or in some other form? It's curious because gawk cannot for one second claim something like needing to stick to some legacy standard, not with a straight face.
  - Joeri 3y ago
    If I were the gawk maintainer I would be unwilling to take on features by default. It is widely used infrastructure and keeping out bugs is far more important than taking on features. If the userbase keeps asking for the same feature over and over, at that point it would be up for consideration, but no sooner.
    - vidarh 3y ago
      I think the point is that the list of features added in gawk vs. POSIX awk is miles long... They've not exactly shown restraint. That said, proving it's value in a fork first seems reasonable.
  - im3w1l 3y ago
    Programming languages try very hard to be backward compatible so every feature you add is an eternal commitment.
  - 3y ago
whartung 3y ago
Funny, I would have approached this by removing the while loop and the if else parts of the BEGIN clause, leveraging the stock file reading and line iteration along with AWK pattern matching (terminated with a next statement to skip to the next row), and then shoved the rest in the END clause. It’s always been a “thing” with me of not liking to put everything into BEGIN. Kind of a “if I’m doing that, why am I using awk” thing. Just how I approach problems with awk.
- eequah9L 3y ago
  I'm with you on this, but is there a way to force input file to read in AWK? That would be a reason to choose the while loop over AWK's implicit iteration. In particular, overriding FILENAME in BEGIN does not do anything.
mtillman 3y ago
I wan’t aware that an updated book was on the way. Pre-ordered it immediately. It's wonderful to see that, even after 40+ years, the people who created a scripting language are still providing new, well-documented features.
jph00 3y ago
awk and sed are cool, but whenever someone tells me they're interested in learning them, I always redirect them to learn perl's `-n` and `-p` flags instead, particularly with `-la` added. This gives you, basically, a superset of sed and awk, which makes many things easier to express, often resulting in clearer and more concise code. For those who have taken this advice, they've always told me later they're really glad they did so, and generally express surprise that this isn't more widely known. (If you already know awk and sed well, then you mightn't view learning perl in addition worth the effort -- I'm not sure either way. This advice is for people that currently are not strong users of either.)
- mattrighetti 3y ago
  Is perl as widespread as sed and awk in linux distros and other OSes? If I want to make a script that works across the board I feel like the latter are much more adopted, is that correct?
  - asicsp 3y ago
    Perl would be better suited as a portable solution, since you only have to take care of feature differences between Perl versions. For grep/sed/awk, you also have to worry about implementation differences (GNU/BSD, gawk/mawk/nawk and so on).
  - mbivert 3y ago
    Perl is essentially a standard on UNIX. In particular, there's a Perl interpreter backed with OpenBSD's base installation[0]. If you avoid recent features (you wouldn't be missing too much), then your Perl code should run with little trouble. However, in my experience, when I start to feel the need to use something more sophisticated than sh/sed/awk, I tend to shy away from Perl in favor of more "robust" languages. Go often is a good-enough substitute (static typing, single-file deployment, trivial cross-compilation); YMMV. [0]: https://marc.info/?l=openbsd-misc&m=159041121804486&w=2 https://marc.info/?l=openbsd-misc&m=159041121804486&w=2
- vidarh 3y ago
  Ruby similarly has -n and -p that does much the same, and Ruby has many of the same `$` vars as Perl/Awk. But even so I often find myself reaching for awk for the simplest stuff because I know it'll be there on pretty much every machine. Only for 1-2 liners, typically though, the moment something grows beyond that, I don't use Awk any more, so really I only use a tiny sliver of what it can do.
greyman 3y ago
I must admit that I use awk only via GPT-4, which will write me the one-liner I need and I just run it. I somewhat cannot remember the syntax, provided I use the tool only occasionally.
teo_zero 3y ago
> return 1 to the caller to indicate we did make an update. This (both in awk and python code) seems useless, as the return value from update() is not used anyhere. Am I missing something obvious?
- benhoyt 3y ago
  Yeah, it's used in the top-level update() call in the BEGIN block: if (update(ARGV[1]) == 0) print ARGV[1] " is up to date"
  - teo_zero 3y ago
    Right, I missed that one, probably considered it a cosmetic feature!
csours 3y ago
Is there a way to get Awk to emit a non-terse version of the script passed in? ie awk '/test/' -> '{ if($0~/test/){print $0} }'
teo_zero 3y ago
I know that these programs are only for didactic purposes and my comment may seem nitpicking, but I can't help noticing that the age comparison of the two versions differ semantically: AWK version uses greater-or-equal, but Python uses strictly-less-than. The behavior is different when target and prerequisite have exactly the same age/mtime: AWK will execute the commands, Python won't. Python's behavior seems wrong to me. It shows up in rules with a phony target and phony prerequisites, which by definition share the same age (9999) and mtime (0). For example, it wouldn't delete prog in the following rule: clean: clean-objs rm prog clean-objs: rm *.o On the other hand, the AWK version has a subtle bug in that it sets to zero the age of a newly updated target: this is not required in the most common cases (because the target will likely be the first file listed by "ls -t" anyway) and makes it incompatible with GNU make in those rare cases when the commands don't actually touch the target. I know they're rare, but just imagine a rule that uses rsync to replace a file with a copy fetched from a remote site only if a newer version exists on that site. If rsync does not download a new version, there's no need to artificially assume that the file was changed, and propagate "upwards" the need to recompile everything that depends on it. Both bugs are easy to correct, though. That could be left as an exercise for the reader!
forgotpwd16 3y ago
Very good article. Enjoyed how it both explored the program and made a similar Python port. I find awk nice for one-liners but, even if interesting more complex programs can be written in it, I prefer the Python version. Worth mentioning the book had some more such programs, like a simple rdbms and a calculator.

260 points by nalgeon 3y ago

I'm a big awk fan but I'm not sold on this. The awk program is not very readable- I think that's fine for a dense one-liner, I'm not really sure it carries over to a 60 line script. I think for something like this I'd prefer a bash script, maybe with awk invoked somewhere, that would be much easier to understand at a glance. Is there something in the awk script that makes it advantageous over a shell script? Edit: I hadn't read the author's conclusion yet when I posted, he agrees I consider AWK amazing, but I think it should remain where it excels: for exploratory data analysis and for one-liner data extraction scripts

Avshalom 3y ago
Well I think generally a 60 line program fits in that spot of "write once, read never, start from scratch if it ever turns out to be inadequate" ... also known as the APL Zone
- yjftsjthsd-h 3y ago
  I'm not dead set against it, but if there was any mistake or bugs I don't know how you'd find them and fix them in that approach
  - fuzztester 3y ago
    By checking the correctness of the outputs, which you need to do anyway?
    - edgyquant 3y ago
      The point is the “and fix them”
      - deleted 3y ago
        [deleted]
    - macintux 3y ago
      Not only is fixing more difficult, but also looking for likely weaknesses (and thus the inputs and outputs to focus on for testing).
    - yjftsjthsd-h 3y ago
      Okay, so the first dev writes 60 lines of indecipherable code, runs some sample invocations, looks at the output, says it looks good. A few months later, someone - maybe the original dev, maybe some other sucker - notices that in some edge case the code misbehaves. Now what? (Obviously, any answer that involves "don't write code with bugs" or "write perfect tests" is a nonstarter)
      - HALtheWise 3y ago
        If we're going with the "start from scratch if it ever proves inadequate" philosophy, then the person who notices the misbehavior looks at the original code, sees that it's written in some obscure language, is undecipherable, but also is only 60 lines long, and decides that it will probably be simpler to make a new (short) implementation in their own favorite language that correctly handles both the original use case and their new requirement. The key insight is that given how much easier it is to write fresh code than understand old stuff, they could very well be correct in that guess, and the end result is a single piece of small clean code, rather than a simple core with layers of patches glued on top. In this particular case, we're talking about a "make" replacement, so testing the new implementation can be done by simply running "make all" for the project. If it passes, then the new implementation must be identical to the old one in all the ways that actually matter for the project at hand. In all likelihood, for a simple program like this, fixing one bug will also silently fix others because the new architecture is probably better than the old one.
        falcor84 3y ago
        I actually really like this approach, and have been thinking about this in regards to coding with an LLM - for a sufficiently simple program (and assuming no security concerns), once you trust your test suite, you should trust AI generated code that passes it. And then if requirements change, you should be able to amend the test cases, rerun the AI until it passes all tests and linters, maybe give the code a quick glance, and be on with your life.
Spivak 3y ago
> for exploratory data analysis and for one-liner data extraction scripts I think both you and the author just don't like AWK if that's the takeaway. What you're describing is literally 1% of the AWK language -- like you don't have to like it, it's weird in many respects but you're treating AWK like it's jq when it's actually closer to like a Perl-Lite/Bash mix. An AWK focused on just those use-cases would look very different. One of my favorite resources on AWK: https://www.grymoire.com/Unix/Awk.html https://www.grymoire.com/Unix/Awk.html
Someone 3y ago
> The awk program is not very readable What do you find hard to read about it? If you know what make does, I think it is fairly easy to read, even for those who don’t know awk at all, but do know the Unix shell (to recognize ‘ls -t’) and C (both of which, probably the audience for this book knew, given that the book is from 1988) > I think for something like this I'd prefer a bash script But would it be easier to read? I doubt see why it would.
- mauvehaus 3y ago
  Bash also would have been an unlikely choice for a book published in 1988, considering it wasn't released until 1989 (Per Wikipedia).
  - Brian_K_White 3y ago
    It would have been ksh, which was the bash of the day, as in, the more featureful sh-compatible sh-superset. But a bash or ksh script would have been less readable than awk. bash (or ksh88 or ksh93) is powrful and useful but not readable if you're actually using the powerful useful features. In bash, a lot of functionality comes in the form of brace expansions and word splitting, basically abusing the command parser to get results there is no actual function for. In awk and any other more normal programming language, those same features come in the form of an explicit function to do that thing.
    - fuzztester 3y ago
      >In bash, a lot of functionality comes in the form of brace expansions and word splitting, basically abusing the command parser to get results there is no actual function for. In awk and any other more normal programming language, those same features come in the form of an explicit function to do that thing. Right. That's one of the reasons why the man page for bash is so long. IIRC, going way back, even the page for plain sh was long, for the same reason.
      - falcor84 3y ago
mbivert 3y ago
I think it should be appreciated in context: it's a good way to teach both awk(1) and make(1) to someone new to UNIX. It also demonstrates how to use awk(1) for prototyping, which IMO is a good programming habit to "develop": it forces to focus on the essential, and not to unnecessarily overthink.
pushedx 3y ago
Writing this make program in bash would invovle even more difficult to read hacks, as bash also does not support multidimensional arrays.
Brian_K_White 3y ago
A while ago I wrote a program to renumber TRS-80 Model 100 BASIC code in awk. Then re-wrote it in bash (pure bash, no sed/grep/cut/tr/awk/bc/etc), and the two are practically identical. That suprised me just how practically identical they wwere in the end. awk is like a hidden miracle of utility just sitting there unused on every machine since the dawn of time. Normally if you want something to be ultra portable, you write it in sh or ksh, (though by now, bash would be ok, I mean there is bash for xenix), but to get the most out of ksh or bash, you have to use all the available features and tricks that are powerful and useful but NOT readable. 50% of the logic of a given line of code is not spelled out in the keywords but in arcane brace expansion and word splitting rules. But every system that might have some version of bash or ksh or plain sh, always has awk too, and even the oldest plain not-gnu awk is a real, "normal", more or less straighforward explicit programming language compared to bash. Not all that much more pawerful, but more readable and more straightforward to write. Things are done with functions that take parameters and do things to the parameters, not with special syntax that does magic transformations of variables which you then parlay into various uses. Everyone uses perl/python/ruby/php/whatever when the project goes beyond bash scope, but they all need to be installed and need to be a particular version, and almost always need some library of modules as well, and every python script breaks every other year or on every other new platform. But awk is already there, even on ancient obscure systems that absolutely can not have the current version of python or ruby and all the gems. I don't use it for current day to day stuff either, there's too many common things today that it has no knowledge of. I don't want to try to do https transactions or parse xml in awk. I'm just saying it's interesting or somehow notable how generically useful awk is pretty much just like bash or python, installed everywhere already, and almost utterly unused.
rottc0dd 3y ago
Bash would really be bad idea if it is going to use bash stitching so many gnu utils for this kind of job. I once had to rewrite a bash script into awk[1] that is big enough and it made the program more readable and the total time execution came down from 12 mins to less than 1 second. I think maybe the original bash script would have written badly, (each util command will invoke it's own process and it has to piped to others instead of using awk which will be running in a single process). [1] - https://github.com/berry-thawson/diff2html/blob/master/diff2html.sh https://github.com/berry-thawson/diff2html/blob/master/diff2...
camgunz 3y ago
It seems pretty readable to me, in particular the "update" function parses as JavaScript if you fix the implicit string concatenation (template literals or +) and replace the # comments with //. I'm actually surprised JavaScript is so similar to awk; it feels like a descendant language tbh.
tyingq 3y ago
> Is there something in the awk script that makes it advantageous over a shell script? Pseudo multi-dimensional associative arrays for representing the dependency graph of make. This part: for (i = 2; i <= NF; i++) slist[nm, ++scnt[nm]] = $i The way awk supports them is hacky and not really a multidimensional array, but still is better than what you would have to do with bash, because of split() and some other language features. It would be much easier with any scripting language though, Perl for example.
agateau 3y ago
I would find it easier to read with more sensible, non-abreviated variable names.

1vuio0pswjnm7 3y ago

These blog posts and discussion usually pit one language against others and often attempt to restrict a language to some specific context, ignoring that each user's experience, needs and preferences may be different. A more interesting debate would be language-agnostic, such as writing one-liners versus writing lengthy programs. In short, the debate might be something like: What does the computer user prefer more: (a) writing one-liners or (b) writing lengthy programs. Not everyone will have the same answer. Knuth might prefer (b). McIllroy might prefer (a). Assuming one reading this blog post knew nothing about programming languages, it seems to imply Python is not well-suited for one-liners, or at least not comparable to AWK in that context. Perhaps the interpreter startup time might have something to do with the failure to consider Python for one-liners.

benhoyt 3y ago
I don't think Python is very well suited to one-liners, but it's not due to interpreter startup time (20ms on my machine). Rather, it's due to all the scaffolding needed, which AWK provides implicitly: AWK automatically reads input lines and splits them into fields, automatically initializes variables to the type's default value, and has terser syntax for things like regex matching. Consider the following AWK one-liner which, for every input line that starts with a letter, prints the line number and the line's second field: awk '/^[A-Za-z]/ { print NR, $2 }' The equivalent Python program has a ton more boilerplate: import statements, explicit input reading and field splitting, and more verbose regex matching: import re import fileinput inp = fileinput.input(encoding='utf-8') for line in inp: if re.match(r'[A-Za-z]', line): fields = line.split() print(inp.lineno(), fields[1])
- fuzztester 3y ago
  >The equivalent Python program has a ton more boilerplate: import statements, explicit input reading and field splitting, and more verbose regex matching: "awks and pythons"
  - fuzztester 3y ago

kazinator 3y ago

Ben notes that Kernighan regrets the way local variables are handled in Awk. I patched GNU Awk to have a @let extension that gives you scoped locals (usable in functions as well as in BEGIN/END blocks): $ egawk 'BEGIN { x = 3; print x; @let (x = 4, y) { print x } print x }' 3 4 3 @ is used because there is at least one other existing extension which is like that: @include. https://www.kylheku.com/cgit/egawk/about/ https://www.kylheku.com/cgit/egawk/about/ This was rejected by the GNU Awk project, though. I was encouraged to make a fork and give it some kind of different name, so I did that.

Brian_K_White 3y ago
Did they cite a reason that sounded reasonable? Like the particular implementation breaks some design principle they want to stick to or something? Did they suggest it might be acceptable some other way or in some other form? It's curious because gawk cannot for one second claim something like needing to stick to some legacy standard, not with a straight face.
- Joeri 3y ago
  If I were the gawk maintainer I would be unwilling to take on features by default. It is widely used infrastructure and keeping out bugs is far more important than taking on features. If the userbase keeps asking for the same feature over and over, at that point it would be up for consideration, but no sooner.
  - vidarh 3y ago
    I think the point is that the list of features added in gawk vs. POSIX awk is miles long... They've not exactly shown restraint. That said, proving it's value in a fork first seems reasonable.
- im3w1l 3y ago
  Programming languages try very hard to be backward compatible so every feature you add is an eternal commitment.
- 3y ago

whartung 3y ago

Funny, I would have approached this by removing the while loop and the if else parts of the BEGIN clause, leveraging the stock file reading and line iteration along with AWK pattern matching (terminated with a next statement to skip to the next row), and then shoved the rest in the END clause. It’s always been a “thing” with me of not liking to put everything into BEGIN. Kind of a “if I’m doing that, why am I using awk” thing. Just how I approach problems with awk.

eequah9L 3y ago
I'm with you on this, but is there a way to force input file to read in AWK? That would be a reason to choose the while loop over AWK's implicit iteration. In particular, overriding FILENAME in BEGIN does not do anything.

mtillman 3y ago

I wan’t aware that an updated book was on the way. Pre-ordered it immediately. It's wonderful to see that, even after 40+ years, the people who created a scripting language are still providing new, well-documented features.

jph00 3y ago

awk and sed are cool, but whenever someone tells me they're interested in learning them, I always redirect them to learn perl's `-n` and `-p` flags instead, particularly with `-la` added. This gives you, basically, a superset of sed and awk, which makes many things easier to express, often resulting in clearer and more concise code. For those who have taken this advice, they've always told me later they're really glad they did so, and generally express surprise that this isn't more widely known. (If you already know awk and sed well, then you mightn't view learning perl in addition worth the effort -- I'm not sure either way. This advice is for people that currently are not strong users of either.)

mattrighetti 3y ago
Is perl as widespread as sed and awk in linux distros and other OSes? If I want to make a script that works across the board I feel like the latter are much more adopted, is that correct?
- asicsp 3y ago
  Perl would be better suited as a portable solution, since you only have to take care of feature differences between Perl versions. For grep/sed/awk, you also have to worry about implementation differences (GNU/BSD, gawk/mawk/nawk and so on).
- mbivert 3y ago
  Perl is essentially a standard on UNIX. In particular, there's a Perl interpreter backed with OpenBSD's base installation[0]. If you avoid recent features (you wouldn't be missing too much), then your Perl code should run with little trouble. However, in my experience, when I start to feel the need to use something more sophisticated than sh/sed/awk, I tend to shy away from Perl in favor of more "robust" languages. Go often is a good-enough substitute (static typing, single-file deployment, trivial cross-compilation); YMMV. [0]: https://marc.info/?l=openbsd-misc&m=159041121804486&w=2 https://marc.info/?l=openbsd-misc&m=159041121804486&w=2
vidarh 3y ago
Ruby similarly has -n and -p that does much the same, and Ruby has many of the same `$` vars as Perl/Awk. But even so I often find myself reaching for awk for the simplest stuff because I know it'll be there on pretty much every machine. Only for 1-2 liners, typically though, the moment something grows beyond that, I don't use Awk any more, so really I only use a tiny sliver of what it can do.

greyman 3y ago

I must admit that I use awk only via GPT-4, which will write me the one-liner I need and I just run it. I somewhat cannot remember the syntax, provided I use the tool only occasionally.

teo_zero 3y ago

> return 1 to the caller to indicate we did make an update. This (both in awk and python code) seems useless, as the return value from update() is not used anyhere. Am I missing something obvious?

benhoyt 3y ago
Yeah, it's used in the top-level update() call in the BEGIN block: if (update(ARGV[1]) == 0) print ARGV[1] " is up to date"
- teo_zero 3y ago
  Right, I missed that one, probably considered it a cosmetic feature!

csours 3y ago

Is there a way to get Awk to emit a non-terse version of the script passed in? ie awk '/test/' -> '{ if($0~/test/){print $0} }'

teo_zero 3y ago

I know that these programs are only for didactic purposes and my comment may seem nitpicking, but I can't help noticing that the age comparison of the two versions differ semantically: AWK version uses greater-or-equal, but Python uses strictly-less-than. The behavior is different when target and prerequisite have exactly the same age/mtime: AWK will execute the commands, Python won't. Python's behavior seems wrong to me. It shows up in rules with a phony target and phony prerequisites, which by definition share the same age (9999) and mtime (0). For example, it wouldn't delete prog in the following rule: clean: clean-objs rm prog clean-objs: rm *.o On the other hand, the AWK version has a subtle bug in that it sets to zero the age of a newly updated target: this is not required in the most common cases (because the target will likely be the first file listed by "ls -t" anyway) and makes it incompatible with GNU make in those rare cases when the commands don't actually touch the target. I know they're rare, but just imagine a rule that uses rsync to replace a file with a copy fetched from a remote site only if a newer version exists on that site. If rsync does not download a new version, there's no need to artificially assume that the file was changed, and propagate "upwards" the need to recompile everything that depends on it. Both bugs are easy to correct, though. That could be left as an exercise for the reader!

forgotpwd16 3y ago

Very good article. Enjoyed how it both explored the program and made a similar Python port. I find awk nice for one-liners but, even if interesting more complex programs can be written in it, I prefer the Python version. Worth mentioning the book had some more such programs, like a simple rdbms and a calculator.