5 ms·

There are things I've come to dislike and avoid when programming in general: - Avoid programming in strings (especially in Bash, where nested quotes are full o

There are things I've come to dislike and avoid when programming in general: - Avoid programming in strings (especially in Bash, where nested quotes are full of pitfalls) - Avoid magic switches that change behavior (like -F) - Avoid terse or cryptic variable names (like $NF) - Avoid terse and magical syntax (sorry Perl, happy to leave you behind me) - Avoid programs that are hard to read - Avoid programs that are difficult to debug while writing them - Avoid programs that ignore types For these reasons, I prefer to avoid awk for anything except the most trivial of tasks. I think the prevalence of scripting languages and the speed of execution and debugging today has made awk not as necessary as it may have been in the 70s. And as to the first point, I'm aware you can write awk scripts in files, and I feel like if your script has gotten complex enough that you need a file, you're creating something unmaintainable and unreadable that would be better suited in a different programming language. Edit: I should add this article is great and a good introduction to awk, regardless of my personal taste for the tool.

m463 5y ago
I use awk for one-liners, no more. Looking at my command history, I mostly use awk to extract a field like this: <something> | awk '{print $3}' (I know "cut" is supposed to do the same thing, but it was never reliable for me - maybe tabs/spaces?)
- chasil 5y ago
  Here is a GAWK program of mine that implements outgoing SMTP. While not a one-liner, this is much shorter and less tedious than trying to do it in C. $ cat /bin/awkmail #!/bin/gawk -f BEGIN { smtp="/inet/tcp/0/smtp.yourco.com/25"; ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /bin/awkmail to from subj < in print "helo " ENVIRON["HOSTNAME"] |& smtp; smtp |& getline j; print j print "mail from: " s |& smtp; smtp |& getline j; print j if(match(r, ",")) { split(r, z, ",") for(y in z) { print "rcpt to: " z[y] |& smtp; smtp |& getline j; print j } } else { print "rcpt to: " r |& smtp; smtp |& getline j; print j } print "data" |& smtp; smtp |& getline j; print j print "From: " s |& smtp; ARGV[2] = "" # not a file print "To: " r |& smtp; ARGV[1] = "" # not a file if(length(sbj)) { print "Subject: " sbj |& smtp; ARGV[3] = "" } # not a file print "" |& smtp while(getline > 0) print |& smtp print "." |& smtp; smtp |& getline j; print j print "quit" |& smtp; smtp |& getline j; print j close(smtp) } # /inet/protocol/local-port/remote-host/remote-port
  - meltedcapacitor 5y ago
    Cheap fix: the space after MAIL FROM: and RCPT TO: is not standard compliant.
  - jrumbut 5y ago
    That's a beautiful use of the language, it reminds me of some of the awk CGI efforts out there. For example: https://www.gnu.org/software/gawk/manual/gawkinet/html_node/CGI-Lib.html https://www.gnu.org/software/gawk/manual/gawkinet/html_node/...
  - m463 5y ago
    IMHO, that's too big for awk, why not python? for example: #!/usr/bin/python import smtplib from email.mime.text import MIMEText msg = 'hi' subj='read this!' smtp_server='mail.example.com' smtp_from='me@example.com' smtp_to='you@example.com' m = MIMEText(msg) m['To'] = smtp_to m['From'] = smtp_from m['Subject'] = subj s = smtplib.SMTP(smtp_server) s.sendmail(smtp_from, [smtp_to], m.as_string()) s.quit() of course, you seem to think in gawk so if that works for you that's what you should continue doing! by the way, I hacked this example from another script which attached a logfile: with open(arg.logfile) as f: log_contents = f.read() m = MIMEText(log_contents) you can also use: from email.mime.image import MIMEImage from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart and then: m = MIMEMultipart() m.attach(MIMEText('\n\n%s\n\n'%xkcd_img_title)) m.attach(MIMEImage(xkcd_img))
    - chousuke 5y ago
      Your script doesn't even do the same thing. You are importing a library that implements SMTP, which is missing the point. The AWK script doesn't need libraries, so it can actually be useful in places where you have awk but not Python.
- likpok 5y ago
  Consider the input a b Awk will treat it as having two columns (by default), while cut will treat each space as it’s own column. Awk is also a little nicer for whitespace; cut makes specifying the delimiter (with say “-d\ “) a little more vexing.
ChuckMcM 5y ago
I take it you LOVE ada :-) There is a lot of wisdom in the things you avoid, however I would ask one question, "How often do you use it?" For me, the best systems are those that can be wordy and prescriptive but as you get to know them you can use more short hand so they "get out of the way" as it were. A good example of that philosophy is keyboard short cuts. When I'm learning a program I'm happy to pause and sling the mouse around to find the thing I need in the labeled menu stack with an appropriate name which also tells me what the keyboard short cut is for that thing. Then as I get better I can just use the short cut and my workflow gets faster. Once I've internalized the keymap my flow is held up by how fast I can think, not by how fast I can take my hand off the keyboard, move the mouse, click and then put it back on the keyboard. Awk is one of those things that once you internalize what it can do, you can use it for a lot of stuff, and you can do it quickly.
jrumbut 5y ago
The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following: col1,col2,col3 1,2,3 4,"hello, \"world\"",6 "7 buckets",,9 To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant. I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great. I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.
- nickcw 5y ago
  There is an answer to CSV mode a bit further down the page https://news.ycombinator.com/item?id=28708145 https://news.ycombinator.com/item?id=28708145 ...but if your files are CSV, there is a CSV extension for gawk @include "csv" BEGIN { CSVMODE = 1 }
  - jrumbut 5y ago
    Well there you go, for the sake of my pride at least it's an extension. It's funny searches for awk CSV seem to yield a bunch of SO questions where the answers are increasingly cumbersome regexes instead of this extension. Of course, you can't count of this extension being widely installed, but it's great for my own desktop.
    - nmz 5y ago
      that's because the extension only works in gawk. its not portable anywhere else.
- sk5t 5y ago
  'miller' and 'xsv' are pretty good tools for wrangling CSV. (And regexp is kind of a terrible tool for it, too many edge cases.)
deleted 5y ago
[deleted]
throwaway38941 5y ago
I've been doing systems work for 20 years. Here's why most of those things are actually good: - Strings are subtly complex, but strings are not variables. You can assign a string, and later handle it as a variable, and not deal with any of the specifics of string-iness. Likewise, you can take a variable, and later treat it as a string (for loosely or not-typed variables). - Magic switches are not magic, they are options. Virtually every program takes options. Sometimes they impact a lot of things, sometimes a little. Only the context determines how much is "too much". - Terse/cryptic variables allow you to write complex expressions in a compact form. This allows you to read more in a small space, making it easier to reason about or form complex expressions. Human languages are flush with these, as is mathematics. But you have to balance the terse, cryptic and magical with guilelessness, or it becomes a mess. - Terse and magical syntax is, again, a feature, not a bug. Using magical syntax I can do in a few characters what would take me many lines with a traditional language, and as we all know, increased number of lines correlates to bugs, in addition to simply making it harder to grok. - Types aren't ignored, but they may be very loosely enforced. If you want to write a quick program to get something done, typing is a curse. If you want to write a very thorough program, typing is a blessing. In many cases, loosely or untyped programs actually work better than their typed cousins, because they allow for more unexpected behaviors without failing. Failing early and often may be a modern trend, but... it literally means things fail more, and this is often not desirable. Caveats: - Programs that are hard to read do indeed suck, and it takes lots of experience to make some kinds of programs easier to read. But that's not an indictment of the program, it's an indictment of the person who wrote it. We don't indict English when somebody writes a document that's impossible to comprehend. - Interestingly, some of the more popular languages are the worst to debug. Perl is probably one of the easiest languages to debug, not inconsequently because of how good the interpreter is at suggesting to the user what the actual problem was and almost exactly how to fix it.
deleted 5y ago
[deleted]
ehsankia 5y ago
Yep, anything more than 1 line would be a python script. The author vaguely tries to answer that but doesn't really give anything more satisfying than "you can rewrite it in python" and "why not?". And if it's something you don't know and are experimenting with, you can also open ipython and play around with the data (kinda like he's doing throughout the article) and keep your variables between each run without having to write intermediate values to disk or keep piping over and over. Let alone the huge stdlib and 3rd party lib you have access to if needed.
wernsey 5y ago
In my day job I use Java, and all the items you list are good ideas to maintain my sanity. But I sometimes use Awk scripts to quickly analyse a log file to get information you need or to generate a configuration file or something. I think of the strict strongly-typed nature of a language like Java as equivalent to the formal language a lawyer might use in a legal document: There shouldn't be any place for ambiguity or misunderstanding. Programming in Awk, on the other hand, is more like having an informal conversation with an old friend; those cryptic variable names is like the slang words that both of you understand and the magical syntax is like the dialect you share that may sound a bit strange to outsiders. I think there is a place for both approaches in the broad space of computing.

m463 5y ago

I use awk for one-liners, no more. Looking at my command history, I mostly use awk to extract a field like this: <something> | awk '{print $3}' (I know "cut" is supposed to do the same thing, but it was never reliable for me - maybe tabs/spaces?)

chasil 5y ago
Here is a GAWK program of mine that implements outgoing SMTP. While not a one-liner, this is much shorter and less tedious than trying to do it in C. $ cat /bin/awkmail #!/bin/gawk -f BEGIN { smtp="/inet/tcp/0/smtp.yourco.com/25"; ORS="\r\n"; r=ARGV[1]; s=ARGV[2]; sbj=ARGV[3]; # /bin/awkmail to from subj < in print "helo " ENVIRON["HOSTNAME"] |& smtp; smtp |& getline j; print j print "mail from: " s |& smtp; smtp |& getline j; print j if(match(r, ",")) { split(r, z, ",") for(y in z) { print "rcpt to: " z[y] |& smtp; smtp |& getline j; print j } } else { print "rcpt to: " r |& smtp; smtp |& getline j; print j } print "data" |& smtp; smtp |& getline j; print j print "From: " s |& smtp; ARGV[2] = "" # not a file print "To: " r |& smtp; ARGV[1] = "" # not a file if(length(sbj)) { print "Subject: " sbj |& smtp; ARGV[3] = "" } # not a file print "" |& smtp while(getline > 0) print |& smtp print "." |& smtp; smtp |& getline j; print j print "quit" |& smtp; smtp |& getline j; print j close(smtp) } # /inet/protocol/local-port/remote-host/remote-port
- meltedcapacitor 5y ago
  Cheap fix: the space after MAIL FROM: and RCPT TO: is not standard compliant.
- jrumbut 5y ago
  That's a beautiful use of the language, it reminds me of some of the awk CGI efforts out there. For example: https://www.gnu.org/software/gawk/manual/gawkinet/html_node/CGI-Lib.html https://www.gnu.org/software/gawk/manual/gawkinet/html_node/...
- m463 5y ago
  IMHO, that's too big for awk, why not python? for example: #!/usr/bin/python import smtplib from email.mime.text import MIMEText msg = 'hi' subj='read this!' smtp_server='mail.example.com' smtp_from='me@example.com' smtp_to='you@example.com' m = MIMEText(msg) m['To'] = smtp_to m['From'] = smtp_from m['Subject'] = subj s = smtplib.SMTP(smtp_server) s.sendmail(smtp_from, [smtp_to], m.as_string()) s.quit() of course, you seem to think in gawk so if that works for you that's what you should continue doing! by the way, I hacked this example from another script which attached a logfile: with open(arg.logfile) as f: log_contents = f.read() m = MIMEText(log_contents) you can also use: from email.mime.image import MIMEImage from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart and then: m = MIMEMultipart() m.attach(MIMEText('\n\n%s\n\n'%xkcd_img_title)) m.attach(MIMEImage(xkcd_img))
  - chousuke 5y ago
    Your script doesn't even do the same thing. You are importing a library that implements SMTP, which is missing the point. The AWK script doesn't need libraries, so it can actually be useful in places where you have awk but not Python.
likpok 5y ago
Consider the input a b Awk will treat it as having two columns (by default), while cut will treat each space as it’s own column. Awk is also a little nicer for whitespace; cut makes specifying the delimiter (with say “-d\ “) a little more vexing.

ChuckMcM 5y ago

I take it you LOVE ada :-) There is a lot of wisdom in the things you avoid, however I would ask one question, "How often do you use it?" For me, the best systems are those that can be wordy and prescriptive but as you get to know them you can use more short hand so they "get out of the way" as it were. A good example of that philosophy is keyboard short cuts. When I'm learning a program I'm happy to pause and sling the mouse around to find the thing I need in the labeled menu stack with an appropriate name which also tells me what the keyboard short cut is for that thing. Then as I get better I can just use the short cut and my workflow gets faster. Once I've internalized the keymap my flow is held up by how fast I can think, not by how fast I can take my hand off the keyboard, move the mouse, click and then put it back on the keyboard. Awk is one of those things that once you internalize what it can do, you can use it for a lot of stuff, and you can do it quickly.

jrumbut 5y ago

The thing that prevents awk from being a major part of my daily routine is that it (amazingly) has poor CSV support. Consider the following: col1,col2,col3 1,2,3 4,"hello, \"world\"",6 "7 buckets",,9 To get the usual awk experience with this very common file format, exactly the type of thing you want to parse with awk, you first need to install gawk, then use a big FPAT regex that needs to be adjusted for any new CSV variant. I would love to see awk with "CSV mode", where it intelligently handles formats like this if you just pass a flag. I think awk would do well to differentiate itself with excellent 2d dataset parsing functionality, but at least catchup up to the average scripting language would be great. I'm half expecting someone to say "just pass -csv it does what you want" and if so I'll be very excited.

nickcw 5y ago
There is an answer to CSV mode a bit further down the page https://news.ycombinator.com/item?id=28708145 https://news.ycombinator.com/item?id=28708145 ...but if your files are CSV, there is a CSV extension for gawk @include "csv" BEGIN { CSVMODE = 1 }
- jrumbut 5y ago
  Well there you go, for the sake of my pride at least it's an extension. It's funny searches for awk CSV seem to yield a bunch of SO questions where the answers are increasingly cumbersome regexes instead of this extension. Of course, you can't count of this extension being widely installed, but it's great for my own desktop.
  - nmz 5y ago
    that's because the extension only works in gawk. its not portable anywhere else.
sk5t 5y ago
'miller' and 'xsv' are pretty good tools for wrangling CSV. (And regexp is kind of a terrible tool for it, too many edge cases.)

deleted 5y ago

[deleted]

throwaway38941 5y ago

I've been doing systems work for 20 years. Here's why most of those things are actually good: - Strings are subtly complex, but strings are not variables. You can assign a string, and later handle it as a variable, and not deal with any of the specifics of string-iness. Likewise, you can take a variable, and later treat it as a string (for loosely or not-typed variables). - Magic switches are not magic, they are options. Virtually every program takes options. Sometimes they impact a lot of things, sometimes a little. Only the context determines how much is "too much". - Terse/cryptic variables allow you to write complex expressions in a compact form. This allows you to read more in a small space, making it easier to reason about or form complex expressions. Human languages are flush with these, as is mathematics. But you have to balance the terse, cryptic and magical with guilelessness, or it becomes a mess. - Terse and magical syntax is, again, a feature, not a bug. Using magical syntax I can do in a few characters what would take me many lines with a traditional language, and as we all know, increased number of lines correlates to bugs, in addition to simply making it harder to grok. - Types aren't ignored, but they may be very loosely enforced. If you want to write a quick program to get something done, typing is a curse. If you want to write a very thorough program, typing is a blessing. In many cases, loosely or untyped programs actually work better than their typed cousins, because they allow for more unexpected behaviors without failing. Failing early and often may be a modern trend, but... it literally means things fail more, and this is often not desirable. Caveats: - Programs that are hard to read do indeed suck, and it takes lots of experience to make some kinds of programs easier to read. But that's not an indictment of the program, it's an indictment of the person who wrote it. We don't indict English when somebody writes a document that's impossible to comprehend. - Interestingly, some of the more popular languages are the worst to debug. Perl is probably one of the easiest languages to debug, not inconsequently because of how good the interpreter is at suggesting to the user what the actual problem was and almost exactly how to fix it.

deleted 5y ago

[deleted]

ehsankia 5y ago

Yep, anything more than 1 line would be a python script. The author vaguely tries to answer that but doesn't really give anything more satisfying than "you can rewrite it in python" and "why not?". And if it's something you don't know and are experimenting with, you can also open ipython and play around with the data (kinda like he's doing throughout the article) and keep your variables between each run without having to write intermediate values to disk or keep piping over and over. Let alone the huge stdlib and 3rd party lib you have access to if needed.

wernsey 5y ago

In my day job I use Java, and all the items you list are good ideas to maintain my sanity. But I sometimes use Awk scripts to quickly analyse a log file to get information you need or to generate a configuration file or something. I think of the strict strongly-typed nature of a language like Java as equivalent to the formal language a lawyer might use in a legal document: There shouldn't be any place for ambiguity or misunderstanding. Programming in Awk, on the other hand, is more like having an informal conversation with an old friend; those cryptic variable names is like the slang words that both of you understand and the magical syntax is like the dialect you share that may sound a bit strange to outsiders. I think there is a place for both approaches in the broad space of computing.