12 ms·
The Awk Programming Language
- binary_ninja 3y agoAwk has always been a language that I loved but I have struggled to use besides quick jobs for parsing text files. I understand it is meant to be use for exactly that, but the fact that is simple, fast and lightweight sometimes makes me want to do something more with it, but when I start trying to do something besides parsing text I find that it starts becoming awkward (pun intented?).
- usrbinbash 3y ago> but the fact that is simple, fast and lightweight I see awk as a DSL to be honest. Yes, it can be used as a general purpose language, but that quickly becomes, as you say, awkward :D Like many DSLs, it is simple, fast and lightweight as long as it is used for it's intended purpose. Once you start using it for something else, these advantages evaporate pretty quickly, because then you have to essentially work around the DSL design to get it to do what you want.
- snitty 3y agoDSL == Domain Specific Language?
- Rediscover 3y agoYes
- coliveira 3y agoOne simple thing I do with awk is to create a command processor: read one line at a time and do things on my data as a response. This is very useful because you can make your command as powerful as needed and call other unix tools as a result.
- rsolva 3y agoDo you have an example of this that is available somewhere?
- bluetomcat 3y agoIt's a language for creating quick alternative views from line- and column-oriented text streams. That means, take the output of another tool and represent it in a different way.
- asicsp 3y agoI use awk mostly for one-liners and resort to Python when I need more than a few lines of code.
- tripflag 3y agoI have found a handful of unconventional applications for awk -- I once needed a tiny pcm pulsewave generator, and awk was surprisingly decent for the job [1]. Aside from that I've mostly been using it for quick statistics [2], but it quickly moves into perl territory... 1: https://github.com/9001/asm/blob/hovudstraum/etc/bin/beeps#L69-L88 https://github.com/9001/asm/blob/hovudstraum/etc/bin/beeps#L... 2: https://ocv.me/doc/unix/oneliners/#965bfcb8 https://ocv.me/doc/unix/oneliners/#965bfcb8
- kqr 3y agoThis is exactly why I moved from AWK to Perl for these quick jobs a couple of years ago. If you stick to an AWK-like subset, Perl is also simple, fast and lightweight. If you want to grow your scripts (and you have a lot of discipline) Perl – in contrast to AWK – gives you enough noose to hang^W^W^W^Wthe tools you need.
- joeythedolphin 3y agoPerl? Wow. Is that better than bash, python or even nodejs? Why write in Perl over these? Serious question, was propaghandized to hate Perl.
- gpvos 3y agoAbsolutely. It is comparable to python in some ways, but makes it much easier to write quick one-liners using regexes and data manipulation, and to scale those up to real programs. It fills the gap between bash scripts using awk, grep and sed, and C/java/C#. Compared to bash scripting, perl is a real programming language. The documentation and library ecosystem are excellent, backwards compatibility is legendary, yet it supports modern Unicode. The syntax is weird, but try it for a bit, read the man pages, it's not that hard. The OO system is weirder, and I wouldn't make complex class hierarchies in it, but it is usable.
- marttt 3y agoI like how Awk is just a single executable. A single-executable Perl that includes only the core library would be great. There is Microperl [0, 1], but no idea how well it compiles with more up-to-date Perl versions. 0: https://github.com/bentxt/microperl-standalone https://github.com/bentxt/microperl-standalone 1: Original article from 2000 by the author Simon Cozens: https://www.foo.be/docs/tpj/issues/vol5_3/tpj0503-0003.html https://www.foo.be/docs/tpj/issues/vol5_3/tpj0503-0003.html
- throw0101a 3y ago> Perl? Wow. Is that better than bash, python or even nodejs? Why write in Perl over these? It depends on scale. If you have some quick parsing to do, then awk will get you started quickly, but as you expand your experimentation on what you want to extract/manipulate, it may not be easy to add onto the awk beginnings of your "one liner". But if you start with awk-like† syntax but invoking it with Perl, then if you find you have to expand, Perl has more elbow room. The intention is not to 'go big', which those other languages may be better at, but to more easily 'start small'. † IIRC, Larry Wall wanted a utility that had awk/(s)ed-like syntax for text manipulation, just 'with more'.
- PhilipRoman 3y agoI find it pretty nice for writing simple preprocessors. For example I have one which takes anything between two marker lines and pipes it through a command (one invocation per block). Awk has an amazing pipe operator which lets you do something like this: ... { print $0 | "command" } "command" is executed once, and the pipe is kept open until closed explicitly by close("command"), at which point the next invocation will execute it again. The command string itself acts as a key for the pipe file descriptor. And of course, no mention of awk is complete without the "uniq" implementation, which beats the coreutils uniq in every way possible (by supporting arbitrary expressions as keys and not requiring sorted input): !a[$0]++
- aktau 3y agoI had no idea about this "keep the pipe open" behaviour. I thought it would spawn the binary on every print statement and thus didn't consider it in the past. But now...
- DarkNova6 3y agoI don't know about Awk, but I feel the urge to write a library named "ward" for it.
- schoen 3y agoMaybe the person who deals with security issues for an awk implementation could be called the awk ward.
- fgh 3y agoWho wrote the second edition?
- Lyngbakr 3y agoI read a comment on HN the other day by someone reviewing the book and I believe they said it was Brian Kernigan.
- fuzztester 3y agoIt was mentioned recently here in another HN thread that Brian Kernighan is writing it.
- B1FF_PSUVM 3y agoThe lowercase 'bwk' used in the text makes me believe that ...
- asicsp 3y agoHave to wait, as "The book will be available by the end of September" See https://hn.algolia.com/?q=The+AWK+Programming+Language https://hn.algolia.com/?q=The+AWK+Programming+Language for discussion on the first edition Didn't know there was a list of `awk` implementations: https://www.gnu.org/software/gawk/manual/html_node/Other-Versions.html https://www.gnu.org/software/gawk/manual/html_node/Other-Ver...
- siraben 3y agoAwk is awesome! Glad that they are looking to modernize the book. It wasn't really necessary, all the code examples in the original edition of the book still run just fine, although some are somewhat dated, like printing ASCII bar graphs. They also had examples of writing VMs, parsers and interpreters in the book, which run on modern implementations.[0] The language has some quirks. To declare temporary variables, it's common practice to add extra arguments to functions that won't be used. And traversal of associative arrays is implementation-dependent. I'm not sure what the situation is regarding locale and UTF-8 support. EDIT: Looks like Brian Kernighan added Unicode support last year.[1] [0] https://github.com/siraben/awk-vm/blob/master/vm.awk https://github.com/siraben/awk-vm/blob/master/vm.awk [1] https://github.com/onetrueawk/awk/commit/9ebe940cf3c652b0e373634d2aa4a00b8395b636 https://github.com/onetrueawk/awk/commit/9ebe940cf3c652b0e37...
- bluetomcat 3y agoIs there a particular benefit in writing a VM in AWK, placed in a big BEGIN block? Very similar code can be written in Perl or Python. Isn't the strength of AWK in its line-matching capability, being able to pattern-match a line against a block of code?
- siraben 3y ago> Is there a particular benefit in writing a VM in AWK Not really. Later on the book just ran out of line-matching examples to go through and started doing regular programming instead :P. When I actually write AWK code I rely on line-matching and using a variable to handle state.
- pdw 3y agoAt the time, awk was the only scripting language (other than shell) generally available on Unix systems. Perl, Tcl, Python didn't exist yet. So awk was often used for general-purpose programming.
- ufo 3y agoI love telling about that example to my programming language friends. > Hey you should read the AWK book, it even says how to write a VM! > Why would I ever want to use AWK for that? > Well, the input is a text file with one space-delimited instruction per line. > Hmm... You have a point.
- radiator 3y agoThis is good news, because you have to pay a lot for a used copy of the first edition nowadays. I hope the spirit remains the same as in the first edition.
- andrewstuart 3y agoAwk and ChatGPT are best friends.
- ketanmaheshwari 3y agoHow so?
- andrewstuart 3y agoAsk ChatGPT to write your awk scripts - it does a prettyy damn good job at a first pass.
- magarnicle 3y agoYeah this solves the "I don't use it enough to remember it problem". ChatGPT eliminates the first hurdle of using it, so I'm likely to use it more, and then hopefully it will start to stick.
- donatj 3y agoI love awk. It’s everywhere and every time I am writing a shell script and work myself into a corner, awk has been the way out. I know exactly enough to be dangerous and have meant to deep dive for almost a decade.
- coliveira 3y agoawk can be mastered by just reading the man page. The book doesn't take long to read either. Once you understand the simple principles, you can write an infinite number of scripts for all kinds of tasks.
- IggleSniggle 3y agoSee, when I'm writing a shell script interactively and work myself into a corner, I reach for awk, struggle with it for a bit, and then either: 1) succeed, and regret the messiness of the solution or 2) fail, and find a non-awk way to handle it. I really tried to like awk, but its portability hasn't been enough of a feature to raise it above other scripting languages for me. Especially if I'm going to end up in an editor
- jhoechtl 3y agoI love the csv-mode. It obviously takes some time
- proger 3y agoFind and AWK together, a match made in heaven. Thanks for the link.
- lkuty 3y agodo you have some resources regarding the use of awk with find ?
- bsdooby 3y agoCurrently looking @ alternatives (not that I dislike AWK, far from it): Tokay: https://github.com/tokay-lang/tokay https://github.com/tokay-lang/tokay frawk: https://github.com/ezrosent/frawk https://github.com/ezrosent/frawk
- sgu999 3y agoWhat do you think of them? Tokay in particular looks very polished.
- bsdooby 3y agoTBH: no conclusion yet (did not find time ATM to try it out in fill detail)...sorry
- lambertsimnel 3y agoHave you considered tab? https://tkatchev.bitbucket.io/tab/ https://tkatchev.bitbucket.io/tab/
- geophile 3y agoTake a look at marcel: https://github.com/geophile/marcel https://github.com/geophile/marcel
- benhoyt 3y agoI was privileged to be one of the technical reviewers for this book. There's a fair bit of the original content (which is still great), but Kernighan's done a great job with some good restructuring and some significant updates, too. The early chapters are very hands-on, with something of a focus on "exploratory data processing", particularly with CSV files. Big data with AWK, you could say. Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode (parsing files with quoted and multiline fields per the CSV RFC). I'm really glad Arnold Robbins added a robust "--csv" implementation to Gawk, too, because that's really the most-heavily used version of AWK nowadays. I've already got CSV support in my own GoAWK implementation, and I'll be adding "--csv" to make it compatible. I'm really glad this new updated version is coming out!
- Simon_O_Rourke 3y ago> Gawk and awk will soon have a new "--csv" option that enables proper CSV input mode Awesome!!!! Super excited to see this!
- calvinmorrison 3y agoIts a crying shame we never settled on a control character separated text format. There's a ascii control characters for record and field (unit) separators. A bit of user space support for that would have been great.
- galleywest200 3y agoIt is a shame. I have been using tab-separated sheets recently as it allows me to simply not care about almost any possible character in my strings...apart from tabs of course. But those are far less common than commas, and putting strings in quotes 100% of the time looks messy to me.
- calvinmorrison 3y agoWay less common would be using ascii 30 and ascii 31. ascii 29 and you can cram multiple datasets into one file
- ducktective 3y agoAlso watch his recent interview on Computerphile: https://www.youtube.com/watch?v=GNyQxXw_oMQ https://www.youtube.com/watch?v=GNyQxXw_oMQ And: Brian Kernighan adds Unicode support to Awk https://news.ycombinator.com/item?id=32534173 https://news.ycombinator.com/item?id=32534173
- throw0101a 3y agoWith Lex Friedman from ~2 years ago: * https://www.youtube.com/watch?v=O9upVbGSBFo https://www.youtube.com/watch?v=O9upVbGSBFo
- bardak 3y agoHonestly after watching a lot of Kernighan interviews and reading his original book on C he is a very great communicator. I wonder how different the software world would have been without him at Bell Labs. Would Unix and C have become as widely used as quickly?
- hi41 3y agoKernighan is a national treasure and a world treasure!
- kar1181 3y agoOne of the first utilities I had to get to grips with way back was awk, and it serves me well to this day. Best bang for buck investment of time in my entire career. Even today I still use some variant of awk -F(x) '{print $x}'.
- apienx 3y agoThanks for your work! Awk is a rabbit hole. "Dark corners are basically fractal - no matter how much you illuminate, there is always a smaller but darker one." - - Brian Kernighan (quoted in the GNU Awk book)
- shaftoe444 3y agoCan I preorder this?
- dzogchen 3y agoWow, hyped for this. I picked up this little book from my University library once, and it was a fantastic read.
- penguin_booze 3y agoI wish awk had support for addressing a range of fields, like from $1 to $7. `cut` supports it, FWIW.
- mplanchard 3y agoYou can always loop through the fields, but it’s a little messy, especially for one-liners
- penguin_booze 3y agoYes, that's an option. The range lookup is an ergonomic feature. Imagne what it would have been like, if we couldn't do foo[-3:] in Python.
- technofiend 3y agoSeems like the best time to ask since this is an awk thread: if anyone has a line on the original artwork or a source for the awk t-shirt please let me know. From memory it's of a gangly bird jumping / parachuting from an airplane (DC3?) and captioned with awk's infamous catch-all error message: "Awk: bailing out near line one".
- ahalbert 3y agoI love using Awk, the only thing I miss is that it can't handle complex csv files. Does anyone know how to handle quoted CSV strings like > "foo","bar,baz"
- asicsp 3y agoIf quoted string is the only thing you need to handle extra (i.e. no escaped quotes, newlines, etc) and if you have GNU awk: $ echo '"foo","bar,baz"' | awk -v FPAT='"[^"]*"|[^,]*' '{print $1}' "foo" $ echo '"foo","bar,baz"' | awk -v FPAT='"[^"]*"|[^,]*' '{print $2}' "bar,baz" For a more robust solution, see https://stackoverflow.com/q/45420535 https://stackoverflow.com/q/45420535 or use other tools like https://github.com/BurntSushi/xsv https://github.com/BurntSushi/xsv
- poetaster 3y agoI wanted to ask why not the more simple form: echo '"foo","bar,baz","boo"' | awk -F"\",\"" '{print $1}' "foo echo '"foo","bar,baz","boo"' | awk -F"\",\"" '{print $2}' bar,baz echo '"foo","bar,baz","boo"' | awk -F"\",\"" '{print $3}' boo" Realizing that I have to strip the quotes that remain. Edit. formatting. EDit, again, from your link, the following is more terse and too my taste (still needs strips): awk -v FPAT='("[^"]*")+'
- geophile 3y agoI like the idea of Unix pipelines, but I hate all the sublanguages, awk being one of the biggest. I scratched my itch and built my own shell, marcel: https://github.com/geophile/marcel https://github.com/geophile/marcel. I mention this specifically, here, because of the CSV point. Marcel handles CSV, e.g. "read --csv foobar.csv" reads the foobar.csv file, parses the input (getting quotes and commas correct), and yields a stream of Python tuples, splitting each line of the CSV into the elements of the output tuples. Marcel also supports JSON input, translating JSON structures into Python equivalents. (The "What's New" section of marcel's README has more information on JSON support, which was just added.)
- lysium 3y agoThey are planning built-in support for that, see that other comment https://news.ycombinator.com/item?id=36518146 https://news.ycombinator.com/item?id=36518146
- csours 3y agoI FINALLY started learning awk in the past couple weeks. I think I was intimidated because awk can be very terse, and there are some default actions that aren't clear when you first start looking at awk scripts. My other problem is that I want to accomplish things, not learn a tool, and it generally takes me a bit longer than it should to decide to actually learn something and not just hack at it. Is it still worth it to be "the awk guy" at work?
- simmonmt 3y agoyes, because you'll be done with your thing before others figure out how to lay out your spreadsheet. also your solution will be reusable. (based on my experience where people who could've benefited from awk for a one-liner dependably reach for sheets/excel rather than something like python or perl)
- MikeTheGreat 3y agoOk, dumb question: Is the link supposed to link to the actual book (i.e., is the book free and/or open source) or is this just a page of miscellaneous interesting links about the book (which we can pay for, later, when it's published). I was expecting the book, but the page itself says "This page is a placeholder for material related to the second edition of The AWK Programming Language." It's fine if this is a placeholder page (and an awesome excuse to read talk about AWK here on HN :) ) but I want to be sure that I'm not missing the book itself.
- RGBCube 3y agoWhat I understand from the page is that the Second Edition of the book will reside in the page when it is released (the reason why it says it is a "placeholder").
- _ph_ 3y agoI think the page description is quite clear: it contains material related to the book. Not the book itself. So I would guess all downloadable code and perhaps supplementory material.
- getpost 3y agoI know lots of people like awk, but I pretend it doesn't exist. Why? Here's my comment on this from 6 years ago[0], >I used awk until I learned Python (long ago). For me, awk was yet another example of the "worse is better" approach to things so common in unix. For example, if you make a syntax error, you might get a message like "glob: exec error," rather than an informative message. "Worse is better" is probably a good strategy in business and for getting things done, but still, mediocrity and the sense of entitlement that so often goes with carelessness, sickens me. [0] https://news.ycombinator.com/item?id=13457265 https://news.ycombinator.com/item?id=13457265 Long live the Unix Hater's Handbook! (Unix is fine, and so are the criticisms herein. Some of these criticisms have been eclipsed by ongoing development.) https://en.wikipedia.org/wiki/The_UNIX-HATERS_Handbook https://en.wikipedia.org/wiki/The_UNIX-HATERS_Handbook
- pmarreck 3y agoI will bet you $1000 that time spent learning Awk will lead to better results much faster than time spent polluting your privileged user directories with Python's excuse for "dependency management"
- paleface 3y agoI agree entirely! For many python users, it’s the only language they know. Often, they see programming in python, as part of their “identity” - so they’re overly invested in it, to the detriment of other wonderful languages, like awk. I used to code perl myself, back in the day - but I came to appreciate the simplicity of awk, and now it’s one of my favourites. I no longer code perl, as a consequence, as I believe awk to be far more elegant! I wouldn’t have done so, if I was overly invested in being a “perl programmer”.
- ghshephard 3y agoYou are missing out. As a former data engineer/current SRE, I spend my entire day with VSCode/Python/Notebooks/CoPilot banging out python code - but whenever I need to do a complex analysis of a semistructured text file in < 60 seconds, awk is my twitch reflex tool. It can trivially do state transition based on patterns in the file, as well as populate hashes from one file and use them in analysis of the next file in just a few characters. Awk's claim to fame in my world is that it's cognitive activation energy for anyone who has taken the 3-4 hours to learn the language from start to finish (and that's the awesome thing about the language - it really is about 3 hours of concentrated attention) - is essentially nil. You see a bunch of ugly not really structured text 500 MB files that you can't pull into pandas, or easily parse into python dicts? No problem - awk will tear through them for you and get the information you want in < 60 seconds, including the time you took to write your (almostl always single line) of code. That's Awk's sweet spot.
- sigzero 3y agoI am looking forward to this coming out.
- rochak 3y agoI wish I use awk all the time but everytime I use it the knowledge I gain doesn't stick. Could be due to its arcane syntax which is just too hard for me to remember.
- pmarreck 3y agoAwk is old but great, designed to chew through lines of text files with ease, and has great defaults that minimize the amount of awk code you actually have to write to do anything. It's underrated.
- ineedasername 3y agoAmazing, takes me back. ~ One of my first big projects at my first job fresh out of college was using sed & awk to semi-automate the transformation of semi-unstructured data into a database. IIRC I couldn't completely automate because it contained author names, from global naming conventions. (parsing names correctly is deceptively complex) They had somewhat arbitrary #'s of initials ranging from 0-3. Again, IIRC, I could easily accommodate 0 or 1 initial (followed by \.) but trying for more would make the regex I was using too greedy and pull in part of the article abstract. These were scientific books and journals. So I scripted a sed & awk program to detect the possibility of > 1 initials and when that occured, I'd pipe the record into nano for a quick review where I manually inserted the correct \. characters for the initials. It was decades of back-catalogue publications for digitization so I sat there for days, listening to music on an original 1st gen iPod, waiting for my duct-taped kludge of a program to pipe one of thousands of records into a nano session every few minutes. This was on an Apple G4 workstation running OS X, where I earned my real bash scripting chops. It was an awful hack by today's standards, but at the time, accomplishing what was expected to be a 1-year long project in ~1 month, it was seen as nearly miraculous.
- threeio 3y agoI read the first edition so many times as a young kid... AWK was just such a cool name when I would go to the library and grab a book out of the stacks trying to learn something new.
- v3ss0n 3y agoAwk was great at its time, but when you need at write more than 5 lines of awk code please consider using python since 1. It is a lot faster than awk/perl/grep/sed combos 2. Way a lot readable and maintainable 3. More powerful than awk with it's string functionalities 4. Same availability as awk in OSs since last decade
- paleface 3y ago[flagged]
- wandering23 3y ago[dead]