8 ms·
Fascination of Awk
- noloblo 4y agoAwk sed bash and perl are extremely underrated and nearly always beats python for elegance and succinctness of the repeating problems in the daily chores of sys admin
- anthk 4y agoPerl itself replaces awk, sed and bash for scripts. Think about it :D.
- effie 4y agoPerl can do everything, but it does not replace those tools. They are better adapted to their specific domains.
- anthk 4y agoThen you don't know neither awk or sed. Most of awk can be redone in Perl with ease, and for regex matching and replacing with sed, PCRE and Perl were practically made for that. I mean, sometimes for small edits sed it's better, and awk for some tabular based files by using xml2tsv or lots of TSV related tools. But for medium sized projects, Perl it's the obvious tool against something that requieres something similar to awk/sed but more complex data parsing.
- effie 4y ago> for small edits sed it's better, and awk for some tabular based files by using xml2tsv or lots of TSV related tools. That, and shell, sed and awk are standard tools which are easy to learn, every unix user should learn basics of (even a java developer), and this won't change anytime soon. However, this can't be said of Perl - it is a powerful tool, but it became culturally obsolete and deprecated, and most unix users in the present and the future won't be bothered to learn it, when learning more modern languages like Python is a better investment.
- pcw888 4y agoI remember the trend away from Perl, but it's a great language. It just became unfashionable.
- lofaszvanitt 4y agoBash, awk et al = the syntax that can't be remembered, not even when smashed in the head with a tiring iron or forced with a submachine gun.
- stn_za 4y agoIt's easy. You just need to stop thinking of these things as toys... Awk + bash could easily recreate most existing code in a couple of lines
- samuell 4y agoI found the awk syntax to be surprisingly discoverable, once I got the rough structure of scripts. I think the confusing factor with awk is that it allows you to leave out variuos levels of structure in the really simple scripts, meaning that the same scripts you see around will look quite different. E.g. all the following would be the same (looking for the string "something" in column 1, and printing only those lines): '$1 == "somestring"' '$1 == "something" { print }' '($1 == "something")' '($1 == "something") { print }' ... to give a small example. At least this confused me a lot in the beginning.
- wiredfool 4y agoawk '{print $3}' is what cut should have been all along. Specifically, this will give you the third space delimited field, where multiple spaces are coalesced. cut -d ' ' -f 3 will get you whatever is between the 3rd and 4th space.
- hawski 4y agoI can agree on bash syntax being crazy, but certainly not on awk. Awk is very simple, a man page is all you need if you need to find something. Otherwise what's so complex with awk?
- abecedarius 4y agoIf you don't like learning from a man page (understandable), I can recommend a quick read of the Awk book by A, K, and W. It's very short and has fun examples.
- asicsp 4y agoMy rough workflow is: * Can I solve the problem using one-liners (grep, sed, awk, perl, sort, etc)? Or perhaps from within Vim? * Can I glue together one-liners with minimal control flow as a Bash script? * If not, go for Python --- Discussion for https://blog.jpalardy.com/posts/why-learn-awk/ https://blog.jpalardy.com/posts/why-learn-awk/ mentioned at the end of the article: https://news.ycombinator.com/item?id=22108680 https://news.ycombinator.com/item?id=22108680 (420 points | Jan 21, 2020 | 235 comments)
- BiteCode_dev 4y agoSame, but with 2 recent : - can I use ripgrep, fdfind, fzf and choose to do this? - can I ask chatgpt to do this ?
- version_five 4y agoIf I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. If I need a script to generate some output (a regular use case that seems to come up for me is random numbers with some property), I tend to use python. I also use python if I need to do more complicated aggregations or something on tabular data that pandas is better at. Though it's fun to try with `join` and awk sometimes (parsing csv can get tricky). If I need to plot something I tend to use jupyter notebook but it's way more satisfying to use gnuplot, which I mention because it fits naturally into workflows that use shell tools like awk.
- masklinn 4y ago> If I have a text file of csv that I need to do something with, I'll usually start with a shell script and coreutils + sed and awk. You should install xsv.
- version_five 4y agoI'm glad you mention that because I forgot something: I try and stick with only stuff I can find on most computers in order to minimize dependencies. That's a big advantage imo of learning how to use awk and tr for basic data analysis, for example. You can ssh into any server and work without installing anything.
- mcculley 4y agoA long time ago, I built a relatively complex program that managed some other systems in awk. It was really a great fit for the problem and I was, at the time, working in an environment with poor developer tooling. The target systems were heterogeneous and I could not depend on Perl even being available. But awk was guaranteed to be there. The problem was that every time someone else was asked to add any feature to it, they freaked out at the language choice and I had to get on a plane.
- pcthrowaway 4y agoAuthor missed an opportunity to reference this 3D doom-like in awk: https://github.com/TheMozg/awk-raycaster https://github.com/TheMozg/awk-raycaster
- rapiz 4y agoWtf
- pk-protect-ai 4y agoIn 1996 I was paid to teach Unix to a group of customers. As a graduation work for my course they had to write an awk program for budgeting without use of any databases. I'm sure they had a lot of fun and cursed me for this sadistic approach to teaching at that time. However decades later some of them were still thankful they have learned regex and awk back then.
- kmarc 4y agoi do this for my own budgeting in 2023. pdf / csv / excel export from my three webbanks, a bit of pdftotext, or soffice conversion just to pipe to awk to augment it and render properly formatted spreadsheet
- joepvd 4y agoawk is amazing. One pattern I often use is: $ query_something | awk 'generate commands' | sh For larger programs, I wrote and use ngetopt.awk: https://github.com/joepvd/ngetopt.awk https://github.com/joepvd/ngetopt.awk. This is a loadable library for gawk that lets you add option parsing for programs.
- mturmon 4y agoThis can be a very powerful idiom (basically, code generation at the shell prompt). It’s well suited to iterative composition of the commands: I’ll write the query/find part, and (with ctrl P) add the awk manipulations, and then pipe to sh. If it doesn’t have side effects you can pass through “head” before “sh” to check syntax on a subset.
- cpdean 4y agoI'm a huge fan of awk but the "Python vs awk" page this links to [1] shows python code that's almost deliberately atrocious. Take this function the author wrote for converting a list of integers (or strings) into floats def ints2float(integerlist): for n in range(0,len(integerlist)): integerlist[n]=float(integerlist[n]) return integerlist Using `range(0,len(integerlist))` immediately betrays how the author doesn't understand python. The first arg in `range` is entirely redundant. Mutating the input list like this is also just bad design. If someone has used python for longer than a month, you'd write this with just `[float(i) for i in integerlist]`. Further down in the function `format_captured` you see this attempt at obfuscation: freqs=ints2float(filter(None,captured[n].split(' '))[2:5]) Why bother with a `filter`? Who hurt you? freqs = ints2float(captured[n].split(' ')[2:5]) That said, the author's implementation in awk does look pretty clean. I'm just peeved that they straw-manned the other language. [1] https://pmitev.github.io/to-awk-or-not/Python_vs_awk/ https://pmitev.github.io/to-awk-or-not/Python_vs_awk/
- version_five 4y agoLooks like he copy pasted the python version from another forum post, and didn't look at it carefully. I'd suspect it can be made to look a lot cleaner (edit, yes, e.g. by just translating each of the main lines in the awk script to an if statement) I agree with the strawman comment.
- randlet 4y agofloats = [float(v) for v in integerList] or less idiomatic floats = map(float, integerList)
- wiredfool 4y agoExcept that these are different -- the second returns an iterator (at least in python3). You'd need list(map(float, integerList)) for them to be equivalent. (It doesn't matter in a lot of cases, but there are enough edges where it does. Json serialization for one)
- 4y ago
- prosaic-hacker 4y agoOther comments at https://news.ycombinator.com/item?id=35281089 https://news.ycombinator.com/item?id=35281089
- sgarland 4y agoIf you find yourself piping any combination of cut, grep, sed, uniq (and likely others I'm missing) together, you can probably do it all in awk. If you can guarantee usage of gawk, you can add sort to that list (tbf you can also implement any algorithm you want in awk, but arguably at that point you're wasting time) - and it's also worth noting that you can dedupe in awk _without_ having sorted input, albeit at the cost of storing all unique lines in memory. Pipes are great because they enable you to trivially send data between programs, and they're terrible for the same reason. While the execution time on modern computers for the average data size isn't noticeable, on larger datasets or repeated execution, it absolutely is. If you don't have to pipe, don't.
- Rediscover 4y ago> ...dedup in awk... I most recently did that an hour ago, and a few hours ago, pretty much every day. awk '! x[$0]++' foo
- cesaref 4y agoYou can write web servers in gawk: https://github.com/crossbowerbt/awk-webserver https://github.com/crossbowerbt/awk-webserver I know, just because you can doesn't mean you should.
- tyingq 4y agoThat's really writing a web server in socat. Still cheating, since I'm using gawk extensions, but here's an awk webserver that's mostly [g]awk. https://news.ycombinator.com/item?id=22085459 https://news.ycombinator.com/item?id=22085459
- pcthrowaway 4y agoI asked ChatGPT to explain that and "prettify" it, for anyone interested (everything below from ChatGPT): This is an AWK script that serves HTTP requests by listening on port 8080. It defines several functions: 1. `send` function takes in status code, status message, content, content type, and content length and sends an HTTP response with the provided information. 2. `cf` function checks if a path contains `..` and returns 0 if it does, otherwise returns 1. 3. `mt` function determines the MIME type of a file using the `file` command. The script sets the record separator RS and output record separator ORS to \r\n which is the line ending used in HTTP. It enters an infinite loop listening for incoming connections on port 8080. When a connection is established, it reads the HTTP request line by line using `getline` and processes the `GET` request. If no path is provided, the script serves `index.html`. It checks if the requested path is safe using the `cf` function, and if it is a file, reads the file and sends an HTTP response using the `send` function. If the path is not valid or the file is not found, it sends a "404 Not Found" response. gawk '@load "filefuncs" @load "readfile" function send(s, e, d, t, b) { print "HTTP/1.0 " s " " e |& S print "Content-Length: " b |& S print "Content-Type: " t |& S print d |& S close(S) } function cf(x) { split(x, y, "/") for (z in y) { print "FOUND " y[z] if (y[z] == "..") { return 0 } } return 1 } function mt(f) { c = "file -b --mime-type " f r = "" while ((c | getline z) > 0) { r = r z } close(c) return r } BEGIN { # Change to the specified directory if (ARGV[1] != "") { if (chdir(ARGV[1])) { print "Failed to chdir to " ARGV[1] exit } ARGC = 1 } # Set the record separator and output record separator RS = ORS = "\r\n" # Listen for incoming connections while (1) { S = "/inet/tcp/8080/0/0" while ((S |& getline l) > 0) { split(l, f, " ") if (f[1] == "GET") { p = substr(f[2], 2) } if (p == "") { p = "index.html" } stat(p, s) if (cf(p) && s["type"] == "file") { m = mt(p) o = readfile(p) send(200, "OK", o, m, s["size"]) break } n = "<html>Not Found</html>" send(404, "Not Found", n, "text/html" RS, length(n)) break } } }' I added some comments to explain what each section of the code does. Let me know if you have any questions!
- ketanmaheshwari 4y agoI am proud of some serious awk I wrote a few years ago: https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Data-Challenge-4.html https://ketancmaheshwari.github.io/posts/2020/05/24/SMC18-Da...
- deleted 4y ago[deleted]
- klodolph 4y agoAwk is actually amazing as long as you operate within its limitations and core problems that it solves well—it is definitely not limited to one-liners, but it has other limitations. It is just so damn good at the things it is good at, but nobody learns to use it, because learning Awk is inefficient in the grand scheme of things. There are better things you can learn. There is one place where Awk has undeniable superiority—and that is its use in environments where bureaucratic rules prohibit the distribution of programs / code (Perl, Python), but where Awk is permitted.
- pphysch 4y agoA naive AWK solution (extract+transform text) will vastly outperform a naive Python solution in throughput and memory footprint. It's absolutely efficient to learn, because there isn't much to learn.
- klodolph 4y agoA lot of people already know how to write Python or Perl or something, and given that background, the case for learning Awk is more difficult. I mean, I like Awk, but I’ve also been programming for a long time and I’ve amortized the cost of learning Awk.
- pphysch 4y agoMy case for learning AWK is if you spend any significant chunk of time in the terminal (sysadmin, DevOps, data eng). Notwithstanding any other scripting languages you know. CLI world is full of bespoke text interfaces. AWK is the tool for dealing with those programmatically.
- PaulHoule 4y agoIt is a guilty pleasure but I like writing awk scripts that write shell scripts that get piped into sh, for example ls | awk '{if (length($1)==7) {print "cat " $1 }}' | sh it is something you really aren't supposed to do because bad inputs could be executed by the shell. Personally the control structures for bash never stick in my mind because they are so unlike conventional programming languages (and I only write shell scripts sporadically) so I have to look them up in the info pages each time. I could do something like the above with xargs but same thing, I find it painful to look at the man page for xargs. When I show this trick to younger people it seems most of them aren't familiar with awk at all. For me the shell is mostly displaced by "single file Python" where I stick to the standard library and don't pip anything, for simple scripting it can be a little more code than bash but there is no cliff where things get more difficult and I code Python almost every day so I know where to find everything in the manual that isn't on my fingertips.
- koolba 4y agoThat’s not so terrible if you at least verify the output before the final “| sh”. Though you’d have to be confident that running it twice is going to give the same results. If it’s remote data that could change then weird/bad/nasty things could happen. For anything non trivial, best to separate those steps and generate a temp script to execute.
- Rediscover 4y agoDo You ever use awk's "system" command? I run stuff like You mentioned (piping to a shell) and also system() frequently. It depends on many factors which one I'll choose. (FWIW, I'm also quite decent in many shell flavors on many Unix/Linux variants, so that is another determinant) Eg, the Busybox ash(1) that I frequently work with does not support arrays, but its awk(1) does...
- john-tells-all 4y agoI love awk! It's incredibly clean and direct. Once I wrote a 300-line Awk script to install a kernel driver. It would scan a hardware bus, and ask the user questions before loading the driver onto the system. Lots of fun!
- jjuliano 4y agoNot until you used JQ inside awk
- xp84 4y agoThe existence of things like "git implemented in awk"* serves as a great reminder that there are a lot of developers out there who are far, far, far, far more talented than I will ever be. I salute them. *Especially keeping in mind that these people wrote things like this for fun.
- evadk8 4y agothe problem with awk is it is slow: counting lines of a json file (some 18mil lines) takes a few seconds where sed is much faster