18 ms·
GoAWK: an AWK interpreter written in Go
- AstroJetson 8y agoWhile I'm cool with writing other language processors in a new language (Lisp written in Cobol anyone?) I'm missing the value of this past the bragging rights. There was a similar article about writing the LuaVM in Go, to package it in bigger Go applications. I've done lots of C based systems and bolted Lua on, so the Go version makes sense. But is imbedding Awk into a program that gets done on a regular basis?
- f2f 8y agoThe performance optimization part of the article is very relevant to experienced Go programmers. Also, pjw, the W in AWK is on the Go team at Google and Kernighan, the K, wrote the canonical Go book.
- bitofhope 8y agoAnd Aho, the A co-wrote the most famous book on compilers. I like to think he'd approve of implementing programming languages.
- 101km 8y agoI believe in this instance it is the author wanting to level up on AWK and Go. The value is learning and fun. An AWK interpreter written in Go is unlikely to be an improvement, except, well here is another blog post you might be interested in that has a similar sense of adventurous tinkering (it's about improving on grep): https://ridiculousfish.com/blog/posts/old-age-and-treachery.html https://ridiculousfish.com/blog/posts/old-age-and-treachery.... That's from 2006 and the tl;dr was graybeards did things a certain way for a reason. And yet nowadays with have things like rg (and ag and a bunch of others).
- bhengaij 8y agoI think my GP has an objection to it being shared and being on top when there is nothing to learn from this in terms of ideas (which I share) and not to people hacking away.
- Insanity 8y agoBut now other people who are interested in Go can learn from it. Seems pretty much the point of HN - finding interesting things to learn.
- cicero 8y agoI look forward to reading this article when I have the time because I am a Go newbie, so a well explained example program will be useful to me. Also, I find language implementation interesting, and I've always admired the AWK language. I for one and glad he did it and wrote an article explaining it. Finally, I think it is good to see C software being rewritten in more robust languages like Go and Rust, if people are inclined to do so. I don't think rewriting all the C software should be our top priority, especially high-quality programs like AWK, but it's a good overall direction to go when people are interested.
- bsg75 8y agoThis might be an avenue for programmers who are comfortable in Go but not C to extend a version of the Awk language. For example I have wished for a long time that Gawk could parse gzipped files so that filenames could be used directly. I could take a run at implementing that in this interpreter where the C version would be more difficult personally.
- t-3 8y agoIs there a reason you can't just use a pipe and process stdin?
- deleted 8y ago[deleted]
- bsg75 8y agoThat is what I do today. However in the case where metadata is part of the file name, the FILENAME variable is not populated without something else in the pipeline that passes it into the Awk script as a variable.
- kris-s 8y agoAWK is such a cool little tool, great way to process log files (as shown in the article). Glad to see it still getting some attention. It's kinda nuts how far the unixy idea of "just streams of text" has gotten us.
- yolo1 8y agoIt's kinda nuts how far the unixy idea of "just streams of text" has gotten us. As much as I agree that the tooling is, object-based tools would be much nicer. for i in `ls`; do echo $i.size; done; versus ls -la | awk '{print $5}' I know which I'd prefer.
- unixhero 8y agoWait a minute. Bash has nice operators like .size ? Omg
- stevekemp 8y agoNo, that was the point of the parent - that it would be nicer if we did have some object-layer, rather than processing (unstructured) text in pipelines. Losing meta-data at each step.
- kccqzy 8y agoSounds like you will like PowerShell.
- TheJoYo 8y agoparty hard.
- luckylittle 8y agoI am hoping one day there is going to be an operating system, where every single tool is written in Go and everything runs in containers! Well done!
- hugelgupf 8y agohere are some basic linux tools in Go: https://github.com/u-root/u-root https://github.com/u-root/u-root
- xcbhdfgh 8y agoHere are some more: https://github.com/ericlagergren/go-coreutils https://github.com/ericlagergren/go-coreutils
- richard_todd 8y agoI wrote a sed in go a few years back, just to do it. The engine is a Reader so you can use a sed-processed stream anywhere a Reader is accepted. There’s also a command-line driver of course. https://github.com/rwtodd/Go.Sed https://github.com/rwtodd/Go.Sed
- Twirrim 8y agoWhy? GC is overkill for a good number of command line tools. What advantage do you see in command line tool running in its own container? Given how important pipes are, that's going to be a lot of overhead punting data between containers.
- dymk 8y agoActually, lack of a GC is overkill (in terms of control needed over memory) for most command line tools. Having to manually track memory liveness C adds a large amount of complexity to tools like awk, sed, grep (which are already complex beasts themselves).
- zzo38computer 8y ago
- snug 8y agoshould have just named it gawk tbh
- jim_bailie 8y agoLua and AWK. I'm inspired! I'm going to rewrite the Perl 5 interpreter in Go. Take that Larry. Just kidding. I'm all for scratching an itch, but why rewrite all these well established tools?
- stevekemp 8y agoI extended somebody elses programming language recently, then wrote a BASIC. Mostly to make sure that I understood lexing, parsing, and AST stuff. While you're right this is reinventing the wheel it can make sense to reimplement old tools to improve safety, security, and to allow them to be embedded in new environments. Have you ever run a fuzz-tester against (GNU) awk? I have. Even now you can segfault awk with bogus programs, for example: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816277 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=816277 No doubt this new implementation won't be perfect, but segfaults should be ruled out via rust/go/ML implementations..
- jim_bailie 8y agoHa! My interaction with AWK throughout the years has been a fuzz test in progress. Very interesting link though. On the other hand, I think what you're saying about expanding a well known tool's range of use in new environments makes sense.
- _ZeD_ 8y agowhy not?
- benhoyt 8y agoI can't tell you what motivates others, but I know what motivated me with GoAWK: intellectual curiosity, and the desire to learn how to use Go's profiling tools on a non-trivial project.
- jim_bailie 8y agoI suppose I'm jaded. I've been writing software for so long and have been so underpaid for it, maybe I'm envious of your ability to follow through on your intellectually curiosity with such flourish. Anyway, great work. Looks good!
- IloveHN84 8y agoNice to see another implementation, but I still hope the original one will be always kept alive
- pmarin 8y agoIt the one used by all the BSD systems. Recently there is an official Github repository with commits from Brian kernighan. https://github.com/onetrueawk/awk/commits/master https://github.com/onetrueawk/awk/commits/master
- LeoNatan25 8y agoWhy do so many modern projects feel the need to include the language and/or the tech stack used as part of the project name? Is it a type of virtue signaling? Does “Go” or “JS” or “Swift” or “Node” make these project more attractive somehow to an end-user (even if the end-user is a programmer)?
- zzo38computer 8y agoYou can tell the difference GoAWK from the other AWK, I suppose.
- LeoNatan25 8y ago¯\_(ツ)_/¯
- pjmlp 8y agoBack it the day GNOME projects had a G somewhere, while KDE ones had a K, which now can be mixed up with Kotlin ones actually. C++ projects used to add ++ as suffix, for example Rogue Wave Tools.h++ and Motif++ libraries back in the glory days of commercial C++ compilers. So this fashion is already quite old.
- LeoNatan25 8y agoIt existed before but much more prevalent these days than before. Or maybe I wasn’t as cranky or prone to eye rolling 15 years ago =]
- black-tea 8y agoBut those are signalling compatibility with other tools. That's fine. It makes sense. But just imagine if someone posted about the original awk (which happens from time to time) and titled it "awk: a text processor written in C".
- pjmlp 8y agoThing is, we now have this thing that language X is not good for doing Y, so when one does post a software for doing Y in X, and it actually does a good job, it works as marketing material for language X. Programming languages are software products as well, and their customers want to feel they have made the right choices sticking with their options.
- akavel 8y agoTroff + pic + eqn ported to Go would be cool :>
- lbruder 8y agoHoly shit yes please. And grap.
- tyingq 8y agoI wonder if there's a file size + workload at which the coordination overhead of a parallel awk is low enough for an overall performance win.
- simplegeek 8y agoI also really like such posts. I have never written an interpreter. What do experts recommend i.e. how can someone without a formal CS background learn about write an interpreter? Any good sources, articles, or books? Thanks in advance.
- benhoyt 8y agoSure! I also don't have a formal CS background (I studied electrical engineering without doing any programming / CS courses). Bob Nystrom's free online book http://www.craftinginterpreters.com/ http://www.craftinginterpreters.com/ is excellent. There's also the (much older but still enlightening) "Let's Build a Compiler" by Jack Crenshaw: https://compilers.iecc.com/crenshaw/ https://compilers.iecc.com/crenshaw/
- nprescott 8y agoDepending on how involved the language you are interpreting is, you might get by having only read chapter 6 of The AWK Programming Language[0] (linked in the article), which covers "Little Languages", including what it terms an assembler and interpreter. If you are interested in more depth, either Crafting Interpreters[1] (mentioned in the article) or Writing an Interpreter in Go[2] looks promising. I've read more of Crafting Interpreters and really enjoy it, though it isn't yet finished. One of the aspects I really enjoy is that the language is implemented and re-implemented in different languages to gradually introduce lower level concepts. Finally, this one may be a little more "out there" than what you are looking for, but if you are interested in designing a language more than the plumbing of an interpreter Beautiful Racket[3] is really good. caveat: not an expert [0]: https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI... [1]: http://www.craftinginterpreters.com/ http://www.craftinginterpreters.com/ [2]: https://interpreterbook.com/ https://interpreterbook.com/ [3]: https://beautifulracket.com/ https://beautifulracket.com/
- kazinator 8y agoAwk in Lisp as a macro with Lisp AST syntax: http://nongnu.org/txr/txr-manpage.html#N-000264BC http://nongnu.org/txr/txr-manpage.html#N-000264BC * Implements almost all salient POSIX features and some Gawk extensions. * awk expression can be used anywhere an expression is allowed, including nested within another awk invocation. Awk variables are lexically scoped locals: each invocation has its nf, fs, rec and others. * awk expression returns a useful value: the value of the last form in the last :end clause. * can scan sources other than files, such as in-memory string streams, strings and lists of strings. * supports regex-delimited record mode, and can optionally keep the record separator as part of the record (via "krs" Boolean variable). * unlike Awk, range expressions freely combine with other expressions including other range expression. * ranges are extended with semantic 8 variations, for succinctly expressing range-based situations that would require one or more state flags and convoluted logic in Awk: http://nongnu.org/txr/txr-manpage.html#N-000264BC http://nongnu.org/txr/txr-manpage.html#N-000264BC * strongly typed: no duck-typed nonsense of "1.23" being a number or string depending on how you use it. Only nil is false. Recently accepted Unix Stackexchange answer featuring awk macro: https://unix.stackexchange.com/questions/316664/change-specific-part-of-file-via-a-shell-script/316752#316752 https://unix.stackexchange.com/questions/316664/change-speci...