9 ms·

Idiomatic Awk (2010)

132 points by StefanBatory 2y ago

asicsp 2y ago
Seems like the link should be https://backreference.org/2010/02/10/idiomatic-awk/ https://backreference.org/2010/02/10/idiomatic-awk/ instead of https://backreference.org/index.html https://backreference.org/index.html
- StefanBatory 2y ago
  Yes - but I am sure I sent the first link. I'm not sure what has happened then.
  - kencausey 2y ago
    I think that it is this in the head: <link rel="canonical" href="index.html"> I had to stop submitting ISC SANS posts because they similarly have a canonical link in their head which HN, for reasons I'm not sure about, use to 'correct' the URL, errantly.
    - dang 2y ago
      Right you are! I can turn that off for specific domains - if you email the relevant info to hn@ycombinator.com I'll take a look.
      - kencausey 2y ago
        Thanks. I'm not sure it is going to be necessary. I checked the ISC diary site again yesterday and I think they have corrected their 'link rel="canonical"' header. I will try to submit one again when I see an interesting one.
- mananaysiempre 2y ago
  Also, (2010).
  - Lammy 2y ago
    Wait until you find out how old AWK is
- dang 2y ago
  Fixed now. Thanks!
Elfener 2y ago
Technically an empty awk program is an implementation of cat(1) (before it came back from berkeley waving flags, anyway). Of course no awk will run an empty program, so the article's '1' or '"a"' or other truthy value is required.
elteto 2y ago
I wanted to like awk and tried really hard but in the end was disappointed by what I see as unnecessary complications or limitations in the language. For example, it has first class support for regexes but only for matching. You can’t do ‘s/foo/bar’. I also found string manipulation to be cumbersome with the string functions. I would have expected a string processing language to have better primitives for this. And function arguments/variables are just a mess, it’s hard to understand how they came up with that design. It’s also quirky and unintuitive in some places you would not expect. Take the non-working example from the article: awk -v FS=';' -v OFS=',' 1 I expect this to change the change the separator in the output. Period. The “efficiency” argument for why it doesn’t work just doesn’t cut it for me. First, it’s very simple to do a one time comparison of FS and OFS, if they are different then you know you know you _have_ to perform the change, because the user is asking you! If I do this in reality and it doesn’t work I just switch over to sed or perl and call it a day. All in all, perl -eP is a better awk. And for data processing I switched to miller. It has it’s idiosyncrasies as well but it’s much better for working with structured records.
- shakna 2y ago
  Awk does support that kind of pattern matching for replacement...? { gsub(/\;/, ","); print }
- CRConrad 2y ago
  > And for data processing I switched to miller. Damn those far-too-common-word names! Betcha if I googled that, I'd get page after page of results pointing to actor Johnny Miller... Any link? [EDIT:] Naah, sorry, just had to search for "miller language" (and remove the superfluous "academy" from "miller language academy" that Google "helpfully" added), and found: https://miller.readthedocs.io/en/latest/miller-programming-language/ https://miller.readthedocs.io/en/latest/miller-programming-l... . Thanks, interesting!
mananaysiempre 2y ago
Funny how exploiting uninitialized variables is evil in (nonstrict) JavaScript but good in Awk. I agree that’s true, mind you, I just can’t pinpoint what about the languages’ designs makes it work in one case but not the other.
- rusk 2y ago
  Guaranteed initialisation rules I would imagine. A lot of languages don’t do this, or assign null or something. For a language with a narrow scope of application you can this but in more general purpose languages it makes less sense.
  - mananaysiempre 2y ago
    Note Awk does have a special value for uninitialized variables—its distinguishing feature is that it compares equal to both zero and empty string simultaneously. (No official way to spell that value, but you can spell it e.g. “undefined” if you never initialize a variable called “undefined”.) It’s really really close to old JS in that respect, and no wonder, given that Awk was an explicit inspiration for JS. But somehow it works for Awk and not for JS, even at the hundred-line scale that’s common and reasonable for both languages.
    - rusk 2y ago
      I never considered that awk was a precursor to JavaScript but I can certainly see that
      - mananaysiempre 2y ago
        Found it! Wirfs-Brock & Eich, “JavaScript: the first 20 years” (HOPL ’20) [1], §3.1: > The syntax of JavaScript 1.0 was directly modeled after the statement syntax of the C programming language with some AWK inspired embellishments. (the rest of the section is also interesting in that respect). [1] https://dl.acm.org/doi/10.1145/3386327 https://dl.acm.org/doi/10.1145/3386327 (CC-BY open access)
rusk 2y ago
I love awk as a language / framework. If it got an uplift to make it useful for more complex problems it would be an absolute winner for lots of basic data processing tasks. I’ve often written incantations many lines long and even broken out to actually writing awk scripts from time to time. Once the penny drops with it it’s great fun but it’s absolutely useless once your problems get to any degree of sophistication. I typically move into Python at this stage. Perl and Ruby are probably a more elegant fit here but those aren’t rows I want to have. In this day and age, awk really needs CSV (RFC-4180) support and better semantic scoping and library support. I’d also think it would be neat as an embedded language for various data processing platforms but if we haven’t seen it yet I doubt we will ever see it. EDIT support for file formats beyond plain text would also be a winner.
- Rotundo 2y ago
  Good news! Not only are there AWK implementations (for instance GoAWK) that can use CSV, but even the one true AWK does do CSV and Unicode nowadays! https://arstechnica.com/gadgets/2022/08/unix-legend-who-owes-us-nothing-keeps-fixing-foundational-awk-code/ https://arstechnica.com/gadgets/2022/08/unix-legend-who-owes...
- theamk 2y ago
  That's why I never write awk one-liners anymore: if I cannot get it done with cut/sed, I jump to perl right away. After all, it was designed to be AWK replacement, and has features like autosplit (-a) and line-loop (-p) [1] that specifically designed to make porting awk's programs easy. And if you need more than simple string matching, Perl still maintains huge CPAN library of modules, so JSON or Date::Parse is one command away. (Note I still use Python/C++ for real programs; perl is only for one-liners to replace the AWK) [0] https://news.ycombinator.com/item?id=36650120 https://news.ycombinator.com/item?id=36650120 [1] https://perldoc.perl.org/perlrun https://perldoc.perl.org/perlrun
- AstroJetson 2y ago
  > EDIT support for file formats beyond plain text would also be a winner. Can you say some more about this, what kind of file format do you want? JSON? XML?
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
sevensor 2y ago
I’ve been appreciating awk more and more lately as a desktop calculator. There are things dc and bc can’t handle, that awk will cheerfully compete for me. Of all the tools I can expect to be present on a Linux machine, it gives me the easiest way to compute a logarithm in a shell pipeline.
ykonstant 2y ago
This is overall a good guide, especially emphasizing the 'condition {action}' pattern; it is a very elegant and clear construct. However, some of the suggestions lean on the "too clever" and can make the code incomprehensible for the newcomer. For instance, if you will use snippets like `awk '!a[$0]++'`, do make sure to comment their use for your sake and others'.
- jnordwick 2y ago
  Honestly that's not a very crazy pattern at all you should be able to read that.
deleted 2y ago
[deleted]

132 points by StefanBatory 2y ago

Seems like the link should be https://backreference.org/2010/02/10/idiomatic-awk/ https://backreference.org/2010/02/10/idiomatic-awk/ instead of https://backreference.org/index.html https://backreference.org/index.html

StefanBatory 2y ago
Yes - but I am sure I sent the first link. I'm not sure what has happened then.
- kencausey 2y ago
  I think that it is this in the head: <link rel="canonical" href="index.html"> I had to stop submitting ISC SANS posts because they similarly have a canonical link in their head which HN, for reasons I'm not sure about, use to 'correct' the URL, errantly.
  - dang 2y ago
    Right you are! I can turn that off for specific domains - if you email the relevant info to hn@ycombinator.com I'll take a look.
    - kencausey 2y ago
      Thanks. I'm not sure it is going to be necessary. I checked the ISC diary site again yesterday and I think they have corrected their 'link rel="canonical"' header. I will try to submit one again when I see an interesting one.
mananaysiempre 2y ago
Also, (2010).
- Lammy 2y ago
  Wait until you find out how old AWK is
dang 2y ago
Fixed now. Thanks!

Elfener 2y ago

Technically an empty awk program is an implementation of cat(1) (before it came back from berkeley waving flags, anyway). Of course no awk will run an empty program, so the article's '1' or '"a"' or other truthy value is required.

elteto 2y ago

I wanted to like awk and tried really hard but in the end was disappointed by what I see as unnecessary complications or limitations in the language. For example, it has first class support for regexes but only for matching. You can’t do ‘s/foo/bar’. I also found string manipulation to be cumbersome with the string functions. I would have expected a string processing language to have better primitives for this. And function arguments/variables are just a mess, it’s hard to understand how they came up with that design. It’s also quirky and unintuitive in some places you would not expect. Take the non-working example from the article: awk -v FS=';' -v OFS=',' 1 I expect this to change the change the separator in the output. Period. The “efficiency” argument for why it doesn’t work just doesn’t cut it for me. First, it’s very simple to do a one time comparison of FS and OFS, if they are different then you know you know you _have_ to perform the change, because the user is asking you! If I do this in reality and it doesn’t work I just switch over to sed or perl and call it a day. All in all, perl -eP is a better awk. And for data processing I switched to miller. It has it’s idiosyncrasies as well but it’s much better for working with structured records.

shakna 2y ago
Awk does support that kind of pattern matching for replacement...? { gsub(/\;/, ","); print }
CRConrad 2y ago
> And for data processing I switched to miller. Damn those far-too-common-word names! Betcha if I googled that, I'd get page after page of results pointing to actor Johnny Miller... Any link? [EDIT:] Naah, sorry, just had to search for "miller language" (and remove the superfluous "academy" from "miller language academy" that Google "helpfully" added), and found: https://miller.readthedocs.io/en/latest/miller-programming-language/ https://miller.readthedocs.io/en/latest/miller-programming-l... . Thanks, interesting!

mananaysiempre 2y ago

Funny how exploiting uninitialized variables is evil in (nonstrict) JavaScript but good in Awk. I agree that’s true, mind you, I just can’t pinpoint what about the languages’ designs makes it work in one case but not the other.

rusk 2y ago
Guaranteed initialisation rules I would imagine. A lot of languages don’t do this, or assign null or something. For a language with a narrow scope of application you can this but in more general purpose languages it makes less sense.
- mananaysiempre 2y ago
  Note Awk does have a special value for uninitialized variables—its distinguishing feature is that it compares equal to both zero and empty string simultaneously. (No official way to spell that value, but you can spell it e.g. “undefined” if you never initialize a variable called “undefined”.) It’s really really close to old JS in that respect, and no wonder, given that Awk was an explicit inspiration for JS. But somehow it works for Awk and not for JS, even at the hundred-line scale that’s common and reasonable for both languages.
  - rusk 2y ago
    I never considered that awk was a precursor to JavaScript but I can certainly see that
    - mananaysiempre 2y ago
      Found it! Wirfs-Brock & Eich, “JavaScript: the first 20 years” (HOPL ’20) [1], §3.1: > The syntax of JavaScript 1.0 was directly modeled after the statement syntax of the C programming language with some AWK inspired embellishments. (the rest of the section is also interesting in that respect). [1] https://dl.acm.org/doi/10.1145/3386327 https://dl.acm.org/doi/10.1145/3386327 (CC-BY open access)

rusk 2y ago

I love awk as a language / framework. If it got an uplift to make it useful for more complex problems it would be an absolute winner for lots of basic data processing tasks. I’ve often written incantations many lines long and even broken out to actually writing awk scripts from time to time. Once the penny drops with it it’s great fun but it’s absolutely useless once your problems get to any degree of sophistication. I typically move into Python at this stage. Perl and Ruby are probably a more elegant fit here but those aren’t rows I want to have. In this day and age, awk really needs CSV (RFC-4180) support and better semantic scoping and library support. I’d also think it would be neat as an embedded language for various data processing platforms but if we haven’t seen it yet I doubt we will ever see it. EDIT support for file formats beyond plain text would also be a winner.

Rotundo 2y ago
Good news! Not only are there AWK implementations (for instance GoAWK) that can use CSV, but even the one true AWK does do CSV and Unicode nowadays! https://arstechnica.com/gadgets/2022/08/unix-legend-who-owes-us-nothing-keeps-fixing-foundational-awk-code/ https://arstechnica.com/gadgets/2022/08/unix-legend-who-owes...
theamk 2y ago
That's why I never write awk one-liners anymore: if I cannot get it done with cut/sed, I jump to perl right away. After all, it was designed to be AWK replacement, and has features like autosplit (-a) and line-loop (-p) [1] that specifically designed to make porting awk's programs easy. And if you need more than simple string matching, Perl still maintains huge CPAN library of modules, so JSON or Date::Parse is one command away. (Note I still use Python/C++ for real programs; perl is only for one-liners to replace the AWK) [0] https://news.ycombinator.com/item?id=36650120 https://news.ycombinator.com/item?id=36650120 [1] https://perldoc.perl.org/perlrun https://perldoc.perl.org/perlrun
AstroJetson 2y ago
> EDIT support for file formats beyond plain text would also be a winner. Can you say some more about this, what kind of file format do you want? JSON? XML?

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

sevensor 2y ago

I’ve been appreciating awk more and more lately as a desktop calculator. There are things dc and bc can’t handle, that awk will cheerfully compete for me. Of all the tools I can expect to be present on a Linux machine, it gives me the easiest way to compute a logarithm in a shell pipeline.

ykonstant 2y ago

This is overall a good guide, especially emphasizing the 'condition {action}' pattern; it is a very elegant and clear construct. However, some of the suggestions lean on the "too clever" and can make the code incomprehensible for the newcomer. For instance, if you will use snippets like `awk '!a[$0]++'`, do make sure to comment their use for your sake and others'.

jnordwick 2y ago
Honestly that's not a very crazy pattern at all you should be able to read that.

deleted 2y ago

[deleted]