9 ms·
Learn Just a Little Awk (2010)
- j2kun 7y agoWhy is this linking to the wayback machine? This blog post is live on the same website at the same url. https://gregable.com/2010/09/why-you-should-know-just-little-awk.html https://gregable.com/2010/09/why-you-should-know-just-little...
- grzm 7y agoSome people link to archive.org to future-proof against dead urls.
- why-oh-why 7y agoThat’s kinda backwards. One should point to the source and if that is no longer available then fire up the Machine. In this case the archive.org link is a 403 for me while the original works.
- thawaway1837 7y agoI like what Firefox does where if a site 404s it checks to see if there is a Wayback Machine snapshot for it and automatically loads it.
- eternalny1 7y agoFirefox doesn't do that. An extension to Firefox does.
- kazinator 7y agoI suspect you misunderstand some of the point. When you make a posting on HN or anywhere else and cite a URL, that URL could go down years later. Your historic posting then has a dangling URL. You can't go back and edit old postings to fix it. Also, some content changes. The URL is valid, but the cited material is not there. With the archive, you can point to a specific date when that was retrieved.
- cbaleanu 7y agoI get a 403 as well, no point in linking to something unaccessible just because reasons. Gdpr is hard but people in Europe should not be denied access just for the sake of immutable URIs
- johnisgood 7y agoWait what? https://web.archive.org https://web.archive.org is inaccessible to you because you are European? FWIW: I am European, and it works for me.
- OrgNet 7y agoit's kind of rude, it's a bit like Google showing large snippets in the web search results so that users never have to leave google.com
- dang 7y agoYeah, that's bad. It's important to use the canonical URL, among other reasons so the site name shows up correctly. Changed now. ingve is an excellent submitter who doesn't normally do that, so I'm sure there was some reason.
- kuharich 7y agoPrior discussions: https://news.ycombinator.com/item?id=1738688 https://news.ycombinator.com/item?id=1738688 https://news.ycombinator.com/item?id=2932450 https://news.ycombinator.com/item?id=2932450
- kylek 7y agoAwk is a fantastic "swiss army knife", and always included (in some form) on every *nix. The article mentions FS but this is easier to access with -F. e.g. for some quick and dirty CSV munging- awk -F, '{print $2}' Something else I find super useful is the built-in regex matching, e.g.- awk '/ERROR/{print $(NF-1)}' or maybe picking out all 503's from an web server access log- awk '/503/{print}' edit: formatting....wow hn is bad for this
- jolmg 7y ago> awk '/503/{print}' {print} is the default action, I believe. So it would suffice to do: awk '/503/' Though, then you may as well just use grep for something like this.
- kylek 7y agoJust a basic example, I'd definitely use grep for something as simple as that. Only pointing out the pattern (you'll definitely have some heftier stuff in {} if you're going to bother with awk in that case)
- dredmorbius 7y agoThe typical awk case is to print selected fields, or operate on them (count, sum, classify, etc). So a '/<pattern>/ { print }' idiom is fairly common, generally on the way to something more refined.
- pizza234 7y agoFor some time I've been using both Awk and Perl for text processing, based on which offered the most convenient command for a given task. There are two (IMO) glaring omissions in Awk: - using the captured group(s) of a regular expression - slicing the arguments array Mawk supports the first functionality, but it's inconvient. Then at some point, I've found that virtually every task I do in my text processing via Awk can be done via Perl in a comparable way... and much more - including the two functionalities above - so I've ditched awk. The only thing inconvenient is to type something like `perl -lane` as opposed to `awk`, but this can be aliased :-)
- ssivark 7y agoThinking over this, it just (finally) struck me that awk (and sed, grep, etc.) are basically just libraries (more like DSLs), and the shell is just another scripting language with a REPL/notebook environment. When that's the case, what makes them particularly useful compared to some other language with a uniform syntax and possibly better forms of "glue" -- say Python, or lisp, or whatever? Why not allow people to acquire proficiency in just one language, and use it consistently to communicate with their computer? I don't think ubiquity/portability is a compelling answer, since (for many/most people who care about this) it is relatively easy to install the environment of their choice on their personal computing device. BTW, we must also strive to maintain that flexibility in the next generation of computing devices. As an aside, folks who do like the task-focused nature of shell DSLs might appreciate language-oriented programming with Racket.
- mordechai9000 7y agoAwk is part of BusyBox, thus it's present by default on a wide range of embedded systems/systems with limited resources. Some people are comfortable working in sh/sed/awk. I personally don't care for it. I'm not familiar with it because I don't use it very much, and I dislike the lack of consistency and all the gotchas in shell scripting. (I'm talking about sh+sed+awk+other command line tools, not just awk.) But I've done it at times when it's all that's available.
- Annatar 7y agoThere is no inconsistency if one programs in straight Bourne shell from 1978, not using any additional features, and frankly, I've been programming in it for 25+ years and never needed "bashisms". ksh is also extremely consistent and portable. One of the main hidden strengths of AWK is that there is one and only one way how to do something in AWK syntax, so from that point, as long as one doesn't use GNU AWK-isms, it's programming luxury and paradise. Programs written in non-GNU AWK run everywhere (one would have to go out of one's way to use Thompson specific features, for example). As far as I'm concerned, it's a matter of understanding UNIX, not GNU/Linux, but UNIX, so that it is clear what is what and what specifically to avoid (like any GNU features). People who grew up on GNU/Linux often struggle with understanding what's what because they have a huge knowledge gap. The fact that GNU/Linux was their first UNIXoid OS is actually highly detrimental now.
- tempguy9999 7y agoI have, and read, a book on awk (the AWK programming language, by guess-the-initials) and didn't see the point. If I need regexes I have emacs. For CSVs I use excel or LibreOffice. It has been a while so could someone remind me why awk is so fab? Perhaps it's just I've never met the right use case to make me fall in love. Sorry if it's a dumb question. TIA
- tgv 7y agoAWK's primary use case is processing a file line by line. It offers a few well thought out mechanisms to do so, allowing you to script relatively complex operations in just a few lines. At the same time, and acknowledging it's not pretty, it's powerful enough to run an http server (not kidding: https://github.com/kevin-albert/awkserver https://github.com/kevin-albert/awkserver). So if you want/need to write a script that requires more sophistication than cat, join and sort offer, but you don't want the hassle of setting up IO, field splitting, pattern matching in a more complex programming language, etc., AWK is your friend. Some of examples of my own awk scripts: converting csv to json, counting trigrams, splitting text by words or sentences, getting the one line from a complex and long log file that I needed, pre-processing files before compilation, cross-tabulation, rewriting file paths, converting weird file formats and ad-hoc templating.
- the_jeremy 7y agoexcel can't handle millions of lines. Every job (only 2, but still) I've had, I've had a problem at least once where I'm given millions of lines and needed to parse them for something. That being said, I used python in those instances, but I'm willing to believe I could've done it faster and more easily with awk.
- tempguy9999 7y agoYes, this was my feeling too, why not just use python and save the effort of learning another language? Perhaps because you can't assume python will be present in a basic *nix install.
- fian 7y agoAwk predates Excel (and therefore LibreOffice), so the original point was that...there was no GUI spreadsheet application to use instead. 32-bit Excel used to have a limit of around 65 000 rows. 64 bit Excel is just over 1 000 000 rows. Both of these limits are commonly exceeded. Excel requires all the data to be in memory. Awk processes line by line so requires less memory. Awk scripts can easily be incorporated in a chain of other commands. Version comparison is easier with Awk scripts as they are text based not binary (xls) or compressed XML (xlsx) files. Awk can be combined with sed, uniq, sort and other tools as one-liners or in shell scripts for fast and powerful data analysis or transformation. These tools are available by default on my Unix-like operating systems. Nowadays though, scripting languages like Perl (inspired by Awk), Python or Ruby might be used instead of Awk. Awk has been a handy tool for me professionally a number of times when I had large amounts of textual data to analyse. I have used it many times to transform data to pass to Oil and Gas engineers who were unable to handle the raw data in Excel.
- Stratoscope 7y agoI don't write much Awk code any more, but at one time it was my favorite programming language, and not just for little one-liners. A while ago I ran across a couple of my old Awk programs and uploaded them to GitHub. One is an implementation of Mark V. Shaney (the infamous Markov chain program), the other takes source code and prints it out in a nice "two up" landscape format on a LaserJet II, including graphical boxes to replace the ASCII dividers in comments like this: //------------------ // Some comment here //------------------ I wish I had a sample printout, but they are long gone. Awk was the first language I used that had associative arrays built into the language, which made SHANEY.AWK almost trivial to write. Especially compared with implementations I saw in C and other languages of the day. I think LJPII.AWK is a good example of how you can write a fairly substantial program in Awk and have it be (mostly) readable. (It was written for either MKS Awk or Thompson Awk, I forget which - so there are a few things in it that may not be available in a typical Awk today.) https://github.com/geary/awk https://github.com/geary/awk https://en.wikipedia.org/wiki/Mark_V._Shaney https://en.wikipedia.org/wiki/Mark_V._Shaney
- salgernon 7y agoI used to struggle at dtrace until I realized it was just awk for a probes instead of regular expressions.
- llacb47 7y ago403 Forbidden
- forinti 7y agoI use a lot of Perl and occasionally some awk. My experience is that Perl is a lot faster (it really shows when you have millions of lines of text to chew). For example, I compared two scripts that find the longest line in a file: Perl took about a second when awk took 21s.
- rukuu001 7y agoSame. Also, many tasks grew in complexity until I threw away my awk/sed combos and wrote some Perl instead.
- jadia 7y agoI think the traffic is too much. 403 Forbidden.
- jpxw 7y agoA lot of the examples are just reimplenting cut.
- deleted 7y ago[deleted]
- ineedasername 7y agoI've found the combination of sed & awk to be a really powerful duo. One of my earliest development projects circa 2002 was to create a complex awk & sed program that took tons of database blobs with inconsistent semi-structured test from a publishing company and converted it to XML, automatically prompting the user for input (using the Pico text editor) whenever there was something that couldn't be resolved. It saved thousands of hours as part of a metadata digitization project that got their journals into ProQuest. As only a semi-literate programmer at the time, it was significantly easier than trying the same thing with a "traditional" programming language. On occasion I still call to awk withing python code when I need to do a one-off quick & dirty <something> For those interested, if you can find a copy, there'a truly excellent book on awk: "The Awk Programming Language", by Alfred Aho, Brian Kernighan, and Peter J. Weinberger
- yesenadam 7y agoA copy: http://gen.lib.rus.ec/book/index.php?md5=0AD0779BF3602F1EA22734B6EEFCA0EA http://gen.lib.rus.ec/book/index.php?md5=0AD0779BF3602F1EA22... Also Arnold Robbins' books Sed and Awk and Effective Awk Programming are excellent, and also detail the many features added since the latest edition of The AWK Programming Language (1988). http://gen.lib.rus.ec/search.php?&req=awk+robbins&sort=year&sortmode=DESC http://gen.lib.rus.ec/search.php?&req=awk+robbins&sort=year&...
- rapfaria 7y agoBuy the books, do not pirate them.
- xvilka 7y agoAwk and sed can provide a better integration with "modern" formats like JSON/YAML/Protobuf/etc. And with corresponding tools like jq/pq/rq. It would help to keep them popular still.
- thewhitetulip 7y agoI recently learnt awk but had a hard time figuring things out (had been trying to learn it since 6yrs) So I wrote a small guide which is hands on for awk https://github.com/thewhitetulip/awk-anti-textbook https://github.com/thewhitetulip/awk-anti-textbook
- asicsp 7y agoReminds me of a similar article [1] Once you get a hang of how shell constructs like pipes, redirections, globs, variable/command substitutions, etc, the various cli text processing tools like grep, sed, awk, perl, sort, tr, pr, paste, etc are worth to learn at least the basics. They have been through years of use and heavily optimized for performance. Just today, a friend of mine called to ask how to improve a Python script's performance for 5-10 MB text file. After confirming he isn't using any other special modules, I advised to check if the performance improves by implementing it using awk. If anyone's interested in examples based tutorials on cli text processing tools, check out my github repo [2] [1] https://blog.jpalardy.com/posts/why-learn-awk/ https://blog.jpalardy.com/posts/why-learn-awk/ [2] https://github.com/learnbyexample/Command-line-text-processing https://github.com/learnbyexample/Command-line-text-processi...
- msla 7y agoThe GNU Project hosts this essay, "The UNIX Shell As a Fourth Generation Language": https://www.gnu.org/software/nosql/4gl.txt https://www.gnu.org/software/nosql/4gl.txt The concept of language "generations" is mostly lost now, much like how we stopped inventing names for how densely we pack transistors onto silicon once we'd gotten to VLSI (Very Large Scale Integration); the point of the essay is that the shell complete with a userspace is a programming language in itself, and one which is much more programmable and extensible than, say, getting all of your data into a database and using SQL to manipulate it. It's a programmable environment, as opposed to a data silo, where the only tools you have are the ones the silo's implementer deigned to provide.
- marble-drink 7y agoThere is a short talk by Eben Moglen [0] which is only 10 minutes long but jam packed with wisdom. One part which changed the way I think about using computers is how he describes GUI interfaces: as a "point and grunt" or "caveman" interface. Using UNIX, on the other hand, is to use language to communicate with the computer and comes with all the benefits that we have when using it to interact with people. [0] https://youtu.be/uKxzK9xtSXM https://youtu.be/uKxzK9xtSXM
- splittingTimes 7y agoI love awk, grep and sed. But I am wondering: in a professional setup, are there no better tools to analyse and interpret structured log messages out there?
- deleted 7y ago[deleted]
- norswap 7y agoI wonder. Unless you have this syntax (and their semantics) memorized so that it just flies from your fingers, it does seem much easier to me to use whatever your language of choice is (Python, Java, ...) to whip up a script. Sure, awk will be terser, but in my case at least, the bottleneck will not be typing, it will be looking up awk features and syntax. Maybe if you do this every day though. I only have need for things like this once every few months.