9 ms·

I love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but on

I love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but only if it was preceded by some other line. I find this to be really useful for creating one-off linters, for example I made one recently to check all our migration files for CREATE INDEX without CONCURRENTLY on a particular set of very large tables where it would cause issues. Since sql statements can be spread over multiple lines, it was difficult to write a straightforward match, but awk can track state like “I’m in a create statement,” “I’m creating an index,” etc. across multiple lines, which allowed me to cobble together something that has worked well for about a year now.

noloblo 3y ago
Can you share this example of tracking state of sql with awk?
- mplanchard 3y ago
  Sure! I posted a gist here, stripped of anything particular to our company: https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c004c796 https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c...
jmholla 3y ago
One of these days I need to get around to learning awk. In the meantime, I've learned some of the deeper, stateful, features of sed. For instance, you mentioned wanting to only output a line if it was preceded by another. Here's a sed command that does so: sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' > echo -e "PREV\nCURR\nCURR\nCURR\nPREV\nRED" | sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' CURR This uses sed's hold buffer. I'll break it down: sed -n The `-n` tells sed no to print anything out. By default, sed prints out whatever is left when processing. We'll tell it with the `p` command when to do so. sed -ne 'x' `-e` indicates we are specifying one of the scripts sed will execute. The command `x` switches the current line with whatever is in the hold buffer. We'll do this on every line. sed -ne 'x' -e '/PREV/ The next command will only run on lines that contain `PREV`. But, because we've been putting lines in the hold buffer, we'll only execute on lines after `PREV` when it has been switched out of the hold buffer. sed -ne 'x' -e '/PREV/ { ... }' The braces indicate all commands should be run when we see this match. sed -ne 'x' -e '/PREV/ { x; ... }' First, we switch the hold buffer with the line buffer. sed -ne 'x' -e '/PREV/ { x; /CURR/ p; ... }' Then, we only print out the line if it contains CURR. sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' Finally, we switch them back in case there is overlap in our matches. (Give `echo -e "PREV\nPREVCURR\nCURR\nCURR\nPREV\nRED"` a try with this.) All that said, I'm pretty sure the `awk` script is much simpler and more direct, but I wanted to share how one might accomplish this was sed. The time I spent learning this probably would've been better spend on awk, but this tutorial[0], was so good and so easy, it taught me nearly everything I know about sed. [0]: https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html
- mbivert 3y ago
  > One of these days I need to get around to learning awk Plan9's awk(1)[0] man page provides a precise and concise (a few paragraphs) presentation of the core features of all awk implementations. Tutorials bring practical knowledge, but often lack complete and self-contained descriptions of those nifty little tools. [0]: https://man.cat-v.org/plan_9/1/awk https://man.cat-v.org/plan_9/1/awk
  - btschaegg 3y ago
    I still maintain that "The AWK Programming Language" [1] is one of, if not the best programming language book I've read so far. It's short and to the point, has good examples, and cuts most of the usual fluff like "what is a variable?". Its base assumptions are: You know how to program, and you're here to learn AWK. Let's get to it. I dearly wish there'd be more books like it for other languages. [1]: https://archive.org/details/pdfy-MgN0H1joIoDVoIC7 https://archive.org/details/pdfy-MgN0H1joIoDVoIC7
    - mbivert 3y ago
      I don't remember reading it but from the ToC & glimpsing at some examples/exercises, it definitely fits in the "Tutorials" category I was thinking of. It's always delightful to see competent authors demonstrate how much sophistication is achievable in about 100 lines of simple code, by comparison with the "the Dog class inherits from the Animal class" type of examples, or "real-life" codebases. This book is definitely in the former category. > I dearly wish there'd be more books like it for other languages. This all reminds me of a well-known regular expression matcher[0], in about 30 lines of C, featured in "The Practice of Programming"[1]. More generally, even without dedicated books, there are common simple-but-sophisticated type of programs that are great to get to know a language, once you have basic programming skills: standard UNIX tools (cat(1), grep(1), etc.), λ-calculus interpreter, LISP interpreter, raytracer, etc. One can often find online versions serving as "solutions." [0]: https://www.cs.princeton.edu/courses/archive/spr09/cos333/beautiful.html https://www.cs.princeton.edu/courses/archive/spr09/cos333/be... [1]: https://en.wikipedia.org/wiki/The_Practice_of_Programming https://en.wikipedia.org/wiki/The_Practice_of_Programming

noloblo 3y ago

Can you share this example of tracking state of sql with awk?

mplanchard 3y ago
Sure! I posted a gist here, stripped of anything particular to our company: https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c004c796 https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c...

jmholla 3y ago

One of these days I need to get around to learning awk. In the meantime, I've learned some of the deeper, stateful, features of sed. For instance, you mentioned wanting to only output a line if it was preceded by another. Here's a sed command that does so: sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' > echo -e "PREV\nCURR\nCURR\nCURR\nPREV\nRED" | sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' CURR This uses sed's hold buffer. I'll break it down: sed -n The `-n` tells sed no to print anything out. By default, sed prints out whatever is left when processing. We'll tell it with the `p` command when to do so. sed -ne 'x' `-e` indicates we are specifying one of the scripts sed will execute. The command `x` switches the current line with whatever is in the hold buffer. We'll do this on every line. sed -ne 'x' -e '/PREV/ The next command will only run on lines that contain `PREV`. But, because we've been putting lines in the hold buffer, we'll only execute on lines after `PREV` when it has been switched out of the hold buffer. sed -ne 'x' -e '/PREV/ { ... }' The braces indicate all commands should be run when we see this match. sed -ne 'x' -e '/PREV/ { x; ... }' First, we switch the hold buffer with the line buffer. sed -ne 'x' -e '/PREV/ { x; /CURR/ p; ... }' Then, we only print out the line if it contains CURR. sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' Finally, we switch them back in case there is overlap in our matches. (Give `echo -e "PREV\nPREVCURR\nCURR\nCURR\nPREV\nRED"` a try with this.) All that said, I'm pretty sure the `awk` script is much simpler and more direct, but I wanted to share how one might accomplish this was sed. The time I spent learning this probably would've been better spend on awk, but this tutorial[0], was so good and so easy, it taught me nearly everything I know about sed. [0]: https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html

mbivert 3y ago
> One of these days I need to get around to learning awk Plan9's awk(1)[0] man page provides a precise and concise (a few paragraphs) presentation of the core features of all awk implementations. Tutorials bring practical knowledge, but often lack complete and self-contained descriptions of those nifty little tools. [0]: https://man.cat-v.org/plan_9/1/awk https://man.cat-v.org/plan_9/1/awk
- btschaegg 3y ago
  I still maintain that "The AWK Programming Language" [1] is one of, if not the best programming language book I've read so far. It's short and to the point, has good examples, and cuts most of the usual fluff like "what is a variable?". Its base assumptions are: You know how to program, and you're here to learn AWK. Let's get to it. I dearly wish there'd be more books like it for other languages. [1]: https://archive.org/details/pdfy-MgN0H1joIoDVoIC7 https://archive.org/details/pdfy-MgN0H1joIoDVoIC7
  - mbivert 3y ago
    I don't remember reading it but from the ToC & glimpsing at some examples/exercises, it definitely fits in the "Tutorials" category I was thinking of. It's always delightful to see competent authors demonstrate how much sophistication is achievable in about 100 lines of simple code, by comparison with the "the Dog class inherits from the Animal class" type of examples, or "real-life" codebases. This book is definitely in the former category. > I dearly wish there'd be more books like it for other languages. This all reminds me of a well-known regular expression matcher[0], in about 30 lines of C, featured in "The Practice of Programming"[1]. More generally, even without dedicated books, there are common simple-but-sophisticated type of programs that are great to get to know a language, once you have basic programming skills: standard UNIX tools (cat(1), grep(1), etc.), λ-calculus interpreter, LISP interpreter, raytracer, etc. One can often find online versions serving as "solutions." [0]: https://www.cs.princeton.edu/courses/archive/spr09/cos333/beautiful.html https://www.cs.princeton.edu/courses/archive/spr09/cos333/be... [1]: https://en.wikipedia.org/wiki/The_Practice_of_Programming https://en.wikipedia.org/wiki/The_Practice_of_Programming