10 ms·
CLI text processing with GNU awk
- ycombinete 3y agoLast week chat-gpt spat out some Awk for me for a generic linux request. Was quite a pleasant surprise!
- deleted 3y ago[deleted]
- asicsp 3y agoHello! Author here. I am pleased to announce a new version of my "CLI text processing with GNU awk" ebook. Learn the `GNU awk` command step-by-step from beginner to advanced levels with hundreds of examples and exercises. This book will dive deep into field processing, show examples for filtering features, multiple file processing, how to construct solutions that depend on multiple records, how to compare records and fields between two or more files, how to identify duplicates while maintaining input order and so on. Regular Expressions will also be discussed in detail. Links: * PDF/EPUB versions: https://learnbyexample.gumroad.com/l/gnu_awk https://learnbyexample.gumroad.com/l/gnu_awk (free till 31-August-2023) * Web version: https://learnbyexample.github.io/learn_gnuawk/ https://learnbyexample.github.io/learn_gnuawk/ * Markdown source, example files, etc: https://github.com/learnbyexample/learn_gnuawk https://github.com/learnbyexample/learn_gnuawk * Interactive TUI app for exercises: https://github.com/learnbyexample/TUI-apps/blob/main/AwkExercises https://github.com/learnbyexample/TUI-apps/blob/main/AwkExer... Bundle offers: * Magical one-liners (https://learnbyexample.gumroad.com/l/oneliners/new_awk_release https://learnbyexample.gumroad.com/l/oneliners/new_awk_relea...) is $5 (normal price $15) — grep, sed, awk, perl and ruby one-liners bundle * All Books Bundle (https://learnbyexample.gumroad.com/l/all-books/new_awk_release https://learnbyexample.gumroad.com/l/all-books/new_awk_relea...) is $12 (normal price $32) — all my 13 programming ebooks I would highly appreciate it if you'd let me know how you felt about this book. It could be anything from a simple thank you, pointing out a typo, mistakes in code snippets, which aspects of the book worked for you (or didn't!) and so on. Reader feedback is essential and especially so for self-published authors. Happy learning :) --- Previous discussions: * Learn to use Awk with hundreds of examples (https://news.ycombinator.com/item?id=15549318 https://news.ycombinator.com/item?id=15549318) — 478 points, Oct 2017, 116 comments * Show HN: An eBook with hundreds of GNU Awk one-liners (https://news.ycombinator.com/item?id=22758217 https://news.ycombinator.com/item?id=22758217) — 539 points, April 2020, 48 comments
- mbwgh 3y agoThanks for sharing this, it is pleasantly obvious you put a lot of work into this. I especially like the TUI application! Will the web version of this book remain free even after August 31st?
- asicsp 3y agoYou're welcome and thanks for the feedback :) Yeah, the web version is always free for all of my ebooks. And you can find the markdown source on GitHub, for example: https://github.com/learnbyexample/learn_gnuawk/blob/master/gnu_awk.md https://github.com/learnbyexample/learn_gnuawk/blob/master/g... I use `pandoc` to generate the PDF/EPUB versions from markdown. See my blog post https://learnbyexample.github.io/customizing-pandoc/ https://learnbyexample.github.io/customizing-pandoc/ for details.
- swores 3y agoI'm curious to know how many people you get paying for something like your "Magical one-liners", and also whether you've ever experimented with a "choose to pay after" model? I ask because it's the kind of thing that I can imagine finding useful enough to pay $5 (or $15) for, but I can also imagine it being something that contains nothing I don't already have saved in my personal "one liners" file, so I'm not really interested in paying to find out.
- asicsp 3y agoI use Gumroad/Leanpub to sell my ebooks. As far as I know, they don't support the "choose to pay after" model. You can see the number of paid sales for the bundles under the "I want this!" button. When the price is 0, it shows the total of both paid/free users. I started selling ebooks about 5 years back. Where I live, my monthly living cost is just $150. While the first two years of sales were just about enough to cover my costs, the last three years have been much better - I can continue being self-employed :)
- Bjartr 3y ago
- thangngoc89 3y agoI have been using ChatGPT for generating these kind of small CLI like this. My prompts look like this: - use jq to count a nested array "a.b.c.d" - find and delete empty folders using `find` - find and replace text using sed/awk I found that using ChatGPT for these purposes boosted my productivity tremendously.
- deleted 3y ago[deleted]
- dotancohen 3y agoChatGPT is a great time saver for those who already know how to use awk. But it should not be used by those who are unfamiliar. Just an example, I saw someone come up with a great awk line to change some text in a nested directory. He then pasted into bash. Only once the server went down did anybody realize that he forgot to cd into the proper directory and he wiped out not only the server config but also all the user-uploaded data as well. The server config was not version controlled and the user data had not been backed up in almost a week.
- swores 3y agoThat's not really a ChatGPT issue, people pasting in slightly wrong commands (or right commands in the wrong folder) is a tale as old as time - well, as old as linux at least. Short of saying that nobody who's already an expert should ever touch a CLI, the lesson from that story is "be as careful as possible, then be more careful, and also have backups of everything" not "don't use a LLM to help".
- simonw 3y agoYeah, that exact same problem could easily affect someone who spent hours cobbling together the same awk script from Google searches and StackOverflow.
- mistercheph 3y agoWhat “care” do you suggesst that someone pasting in a script they don’t understand should take?
- ymgch 3y agoWhat is better? Starting with awk or sed?
- asicsp 3y agoDepends on the task. Sed is typically used for search and replace and Awk is better suited for field based processing. Both these tools also have filtering features (regexp based, line number based, range, etc). See also: When to use grep, sed, awk, perl, etc https://unix.stackexchange.com/q/303044 https://unix.stackexchange.com/q/303044
- _ZeD_ 3y agoWhile each with their scope and idiosyncrasies ... they're pretty similar, at least in the pattern matching part, and both of them have a pretty internal "core" of functionalities that is easy to grasp. so the honest answer IMHO is ... "both"
- alpaca128 3y agoIf you're familiar with Vim's search/replace syntax you already know how to use "sed -e" to replace text, that's how I got into it.
- chasil 3y agoA sed binary is usually much smaller than an awk binary, either POSIX or GNU. The memory footprint of sed will be much more compact. However, sed has grown out of the command language used by the tty editors, and is more difficult to program (although it is Turing-complete). The awk language implements much of the syntax of C, and it is not difficult to write a very slow and inefficient script. This inefficiency is harder to reach in sed, because it takes more effort to abuse it. O'Reilly's book on sed and awk is available free online, both to browse and to download as a ZIP. https://docstore.mik.ua/orelly/unix/index.htm https://docstore.mik.ua/orelly/unix/index.htm
- Anthony-G 3y agoFor those who care about copyright, that URL is not from O’Reilly; it’s a copy of a book-set that O’Reilly used to distribute via CD-ROM – with a nice user interface that used web technologies (even included a search feature). O’Reilly could make it available for free – as they’ve done so for other books such as Apache Security¹ or Using Samba² – but they still (as is their right) expect you to pay for the sed & awk book³. ¹ https://blog.ivanristic.com/2015/02/apache-security-ten-years-later.html https://blog.ivanristic.com/2015/02/apache-security-ten-year... ² https://www.oreilly.com/openbook/samba/book/ https://www.oreilly.com/openbook/samba/book/ ³ https://www.oreilly.com/library/view/sed-awk/1565922255/ https://www.oreilly.com/library/view/sed-awk/1565922255/
- qalmakka 3y agoAwk is fine and dandy but, like wity Sed, I think that it's almost always replaceable with Perl which is way nicer to use, and ubiquitous. Every OS (except Windows) I laid my hands on in the last 15 years has had Perl installed in either its default install or pulled in as a dependency almost immediately (a LOT of stuff depends on Perl in any Unix system). This is, unless you are running on an embedded environment, but in that case you are stuck with something like busybox's Awk which is way more limited than gawk...
- delta_p_delta_x 3y ago> except Windows On Windows, you use PowerShell.
- layer8 3y agoI prefer Cygwin.
- horse_dung 3y agoI just finished writing a “dumb stuff with containers” internal blog which included: C:\> type somefile.txt | docker run --rm -i ubuntu awk ‘something’ > output.txt
- swores 3y agoThey were talking about whether or not the OS comes with Perl by default, not whether it has a CLI at all.
- thesuperbigfrog 3y ago>> On Windows, you use PowerShell. If you use Git for Windows (https://gitforwindows.org/ https://gitforwindows.org/), it includes Perl. Or you could install Strawberry Perl which is made for Windows: https://strawberryperl.com/ https://strawberryperl.com/
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- swores 3y agoI suspect that anyone reading this thread is likely to be equally interested in "Ask HN: Share a shell script you like" from a fortnight ago (though at 78 comments, it didn't get as much traction / comments as I hoped it would when I saw it) https://news.ycombinator.com/item?id=37112991 https://news.ycombinator.com/item?id=37112991
- asicsp 3y agoThere was a similar discussion 5 months back: https://news.ycombinator.com/item?id=35122780 https://news.ycombinator.com/item?id=35122780 (332 points | 328 comments) And here's one from last year: https://news.ycombinator.com/item?id=32467957 https://news.ycombinator.com/item?id=32467957 (374 points | 294 comments)
- swores 3y agoThanks :)
- tyingq 3y agoOne somewhat not-well-known thing with gawk is that it typically ships with some useful extensions that give you access to things like readdir(), ord(), chr(), gettimeofday(), sleep(), etc. https://www.gnu.org/software/gawk/manual/html_node/Extension-Samples.html https://www.gnu.org/software/gawk/manual/html_node/Extension...
- rottc0dd 3y agoconfession: plug I once wrote a diff2html script ported from bash and it was much, much faster (for obvious reasons). And awk makes it much more readable than bash script. And I could learn the language, debug, understand bugs and fix them in a night. Not sure, if it is idiomatic way to awk, but have to say it is a really nice language. https://github.com/berry-thawson/diff2html/blob/master/diff2html.sh https://github.com/berry-thawson/diff2html/blob/master/diff2...
- e63f67dd-065b 3y agoPerhaps my old sysadmin hat is showing through, but I don’t quite see what the advantage of awk is over just writing the same thing in perl. I’ve seen my fair share of horrendous shell scripts from junior sysadmins, and every time I think to myself “the text processing portion would be so much cleaner in Perl”.
- pletnes 3y agoRemove perl, have less security issues. Some scanning tools flag it, too. Awk is found in more places, in my experience.
- ajross 3y agoSorry, but that's ridiculous. Any general purpose programming language is a vector for bugs and security problems, but come on: you're genuinely trying to say that a kludge of bash+sed+awk is objectively more "secure" than a single perl script to solve the same problem?
- coliveira 3y agoIn the case of awk, actually yes, it is safer. The reason is that awk is a very limited language. It has only enough functionality to provide text matching and substitution. It is very difficult to use awk to do anything of high security risk, compared to a language like perl.
- ajross 3y agoBut awk is never used alone. You don't solve whole problems with awk, you squish it into a script with a bunch of other junk. My point is that you're making an apples-to-oranges comparison. Sure, "awk" isn't the problem, but "bash" is, and bash is undeniably a more error-prone language than perl. You surely agree with that much, right? And if you disallow "bash" for security reasons, where does that leave "awk" in the category of useful tools? See my point?
- cb321 3y agoOf possible interest - instead of making a whole new programming language like awk, you can also just systematize generating code for an existing one with a command-line harness. This can even stay terse & keep a fairly fast edit-test turnaround in a fully statically typed language like Nim: https://github.com/c-blake/bu/blob/main/doc/rp.md https://github.com/c-blake/bu/blob/main/doc/rp.md
- macintux 3y agoWhole new language? Awk is 45 years old.
- cb321 3y agoI agree that "writing/learning a different" (what I meant) is more clear wording than "making a whole new". EDIT: and it is a fair counterpoint that any command with options is, in some sense, also a different language one must learn. Learning API calls is also a different language (at least nouns & verbs if not syntax). But that is all partly the point. awk did/does a programming language with different syntax where other alternatives might be enough.
- BaseballPhysics 3y agoIf you think that's a good idea, you don't understand why tools like awk/perl/sed/etc exist and are popular. They are, by design, optimized toward specific kinds of use cases. In fact, their dynamically typed nature is a perfect example of that since it's much easier to quickly manipulate strings in a language that isn't so strict, as they'll do more heavy lifting for you via automatic coercion while limiting extra syntax/boilerplate (which, granted, is less of a problem with modern type inference). That makes it a lot easier to toss together quick one-liners and glue code, which is where these tools shine in the first place. Hell, even something like python or ruby is just a little too structured for my taste when doing something quick and dirty, which is why I love perl as it can be unstructured if that's all I need, or I can create a more structured program if that's what the problem requires.
- 3y ago
- mpalmer 3y agoA few years back I decided to just get as capable as I could with jq, which is fast and functional enough to cover 99% of awk/sed use cases, plus cases you'd never want to touch with awk/sed. No regrets!
- healeycodes 3y agoAwesome! I've been meaning to replace my usage of Python/JavaScript for tasks (which I believe) are more awk-shaped.
- SOLAR_FIELDS 3y agoAs long as you don’t care about unit test ability. Usually if you bothered to write them in Python or JS you usually don’t want to regress back to shell stuff. You’re already in a place where you have a runtime available so you can do way more stuff. It’s usually the opposite direction that you mentioned that you want to go. You one liner some shell like awk to quickly get shit done without worrying about a runtime being available to you and then if you need it to be more robust and legible because of testing etc or production grade you move to a proper dynamic scripting environment
- Galanwe 3y ago99.9% of my awk use case is to split a line (a la "cut - d\ - f) while discarding successive spaces. e.g.: $ echo "key: value" | awk '{print $1}' value Open to a simpler replacement :-)
- cb321 3y agoYou might consider: https://github.com/c-blake/bu/blob/main/doc/cols.md https://github.com/c-blake/bu/blob/main/doc/cols.md That's in Nim, though that may not be much a barrier. (There may also be other tools in bu/ of interest.)
- llimllib 3y agoYou can do it with cut too: $ echo "key: value" | cut -wf 2 value but whether it's actually "simpler" is open to debate edit: actually gnu cut lacks -w, so this is bsd-only. lol computers, stick with awk
- cb321 3y agoOnly with FreeBSD `cut`.. coreutils `cut` (Linux) is missing -w as is at least OpenBSD? EDIT: I see you discovered this. lol computers indeed. ;-)
- version_five 3y agoWhat does -w do? This works without it, no? Edit, found it, "use whitespace as the delimiter" https://www.unix.com/man-page/FreeBSD/1/cut/ https://www.unix.com/man-page/FreeBSD/1/cut/ For most cases like the OP you'd know the delimiter anyway so I don't think the absence is a big deal, and if not it would be easy to use tr or sed to make it consistent
- llimllib 3y agothe important thing is that it uses _consecutive_ whitespace as a the delimiter, so you'd have to use sed to collapse all the whitespace down to one tab. At that point, awk is vastly simpler
- 3y ago
- nologic01 3y agoawk one-liners are a slam dunk. The tough question whether to invest in more complex awk programming. Invariably some processing task requires more complex logic and awk provides that, but in the terse and arcane ways of early computing. Yet reaching for a modern alternative is also an overhead, may not be particularly intuitive either (hello pandas) and may even have performance issues...
- martincmartin 3y agoFor me, the big problem is libraries. Even a personal file of common functions doesn't seem that well supported, and there just doesn't seem to be a way to get third party libraries. When I start needing helper functions and splitting it into multiple lines is usually when I reach for Python instead. And then sigh, because my program will be 2 to 3 times bigger. Ruby is a great awk replacement, but unless other people at your job know it, you can't expect others to maintain it.
- cb321 3y agoI'd bet you could do a harness like https://news.ycombinator.com/item?id=37292882 https://news.ycombinator.com/item?id=37292882 mentions but with Python in like an hour and then you could stay in both one syntax and more significantly in one library ecosystem. Why, 3 such things may even already exist. :) The syntax/semantics is not as optimized for 1-liner brevity, but everything has trade-offs.
- nmz 3y agoThe problem of libraries is namespaces, since everything is global its not worth it. and also incredibly problematic (especially since match() actually sets a 2 globals) fortunately gawk offers namespaces, it even has a flag for loading libraries from a path { gawk -i inplace } has replaced sed -i for me a couple of times. But yeah, its still lacking. PS: This is gawk only though, but awk -f $awkmodules/mymodule.awk -f <(echo '') is an ok replacement, even though its just concatenating files
- SOLAR_FIELDS 3y agoI used to not like them pre chat gpt. But nowadays when you can paste an arcane awk/sed illegible one liner into an AI and have it describe step by step what it’s doing in totally fine with it now. I still don’t like them as much as a few lines of python for unit test ability reasons but sometimes you just straight up don’t need unit tests for some quick data munging task
- mplanchard 3y agoI love awk, and I find myself reaching for it a fair bit. One of the main things I use it for is “sed with state,” so for things like matching on a line, but only if it was preceded by some other line. I find this to be really useful for creating one-off linters, for example I made one recently to check all our migration files for CREATE INDEX without CONCURRENTLY on a particular set of very large tables where it would cause issues. Since sql statements can be spread over multiple lines, it was difficult to write a straightforward match, but awk can track state like “I’m in a create statement,” “I’m creating an index,” etc. across multiple lines, which allowed me to cobble together something that has worked well for about a year now.
- noloblo 3y agoCan you share this example of tracking state of sql with awk?
- mplanchard 3y agoSure! I posted a gist here, stripped of anything particular to our company: https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c004c796 https://gist.github.com/mplanchard/07229d61bd32ce73624d9003c...
- jmholla 3y agoOne of these days I need to get around to learning awk. In the meantime, I've learned some of the deeper, stateful, features of sed. For instance, you mentioned wanting to only output a line if it was preceded by another. Here's a sed command that does so: sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' > echo -e "PREV\nCURR\nCURR\nCURR\nPREV\nRED" | sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' CURR This uses sed's hold buffer. I'll break it down: sed -n The `-n` tells sed no to print anything out. By default, sed prints out whatever is left when processing. We'll tell it with the `p` command when to do so. sed -ne 'x' `-e` indicates we are specifying one of the scripts sed will execute. The command `x` switches the current line with whatever is in the hold buffer. We'll do this on every line. sed -ne 'x' -e '/PREV/ The next command will only run on lines that contain `PREV`. But, because we've been putting lines in the hold buffer, we'll only execute on lines after `PREV` when it has been switched out of the hold buffer. sed -ne 'x' -e '/PREV/ { ... }' The braces indicate all commands should be run when we see this match. sed -ne 'x' -e '/PREV/ { x; ... }' First, we switch the hold buffer with the line buffer. sed -ne 'x' -e '/PREV/ { x; /CURR/ p; ... }' Then, we only print out the line if it contains CURR. sed -ne 'x' -e '/PREV/ {x; /CURR/ p; x}' Finally, we switch them back in case there is overlap in our matches. (Give `echo -e "PREV\nPREVCURR\nCURR\nCURR\nPREV\nRED"` a try with this.) All that said, I'm pretty sure the `awk` script is much simpler and more direct, but I wanted to share how one might accomplish this was sed. The time I spent learning this probably would've been better spend on awk, but this tutorial[0], was so good and so easy, it taught me nearly everything I know about sed. [0]: https://www.grymoire.com/Unix/Sed.html https://www.grymoire.com/Unix/Sed.html
- auselen 3y agoI did this golfing a while back: Drawing a heart with AWK - https://gist.github.com/auselen/906a53b47a7d616b080dbef85eb8f776 https://gist.github.com/auselen/906a53b47a7d616b080dbef85eb8...
- ra1231963 3y agoNever learned awk or committed esoteric cli incantations to memory. Don’t get me wrong, I can get around on the cli, but sed, awk, etc just didn’t seem like a good cost/benefit investment. I’m also not a sysadmin. Thankfully I waited long enough and LLMs can write them for me better than I ever could.
- deleted 3y ago[deleted]
- rubicks 3y agoI love awk. Enough to shill for this: https://www.oreilly.com/library/view/effective-awk-programming/9781491904930/ https://www.oreilly.com/library/view/effective-awk-programmi... If TFA is an excerpt for a book forthcoming on dead-tree media, then I'll be buying that one as well.
- deleted 3y ago[deleted]
- kazinator 3y agoI maintain a minor side interest in Awk, along side Lisp and other things. I developed cppawk in 2022: https://www.kylheku.com/cgit/cppawk/about/ https://www.kylheku.com/cgit/cppawk/about/ cppawk extends Awk with preprocessing. There is a loop macro that supports a vocabularly of clauses. Clauses can be combined for parallel and cross-product iteration. And they are user-extensible. By writing five simple macros, you can define a new clause. Something potentially useful if you use Awk. Cppawk is documented with multiple man pages, and covered by unit tests which run with gawk and mawk.