15 ms·
Brian Kernighan adds Unicode support to Awk
- ufo 4y agoWhat does adding unicode support mean? Is it about making all the string indexing match codepoints instead of bytes? Or is it more than that?
- sargstuff 4y agoextending ascii using bytes on the fly for type setting bounded in a utf-8 character space.
- lizardactivist 4y agoIs use of tools like awk, m4, sed etc. common and growing, or decreasing? I've always found that for my personal use cases, writing a little Python snippet has always been much faster and simpler than understanding what CLI tool to use, and how to use it, to achieve the same thing.
- MrDresden 4y agoDepends on the situation. I've had to work on environments where awk was available by default, but Python was not (and was not going to be). It can be good to have a screwdriver for every type of screw in the toolbox.
- nmz 4y agoI only use awk. sed is unnecessary when you know awk has gsub or sub. Though I would say, I'd really like lua to be in there considering its small size.
- deleted 4y ago[deleted]
- hi41 4y agoK is a gift to humanity who keeps on giving. I am a big fan of his C book and the AWK book! Long live, K!
- blacklight 4y agoI also became aware of this from Kernighan's interview on Computerphile. And I thought to myself "man, such an omnipresent and fundamental piece of software doesn't have UTF-8 support yet?" and then I thought that, after >20 years of using awk in my script, I never had the urge to use it to parse UTF-8 - but I guess that it's just the luck of living in a country that uses a Latin alphabet, and I couldn't imagine how people could use it on documents in e.g. Chinese or Arabic. This was really an example of how discriminatory software is when it comes to languages.
- dredmorbius 4y agoI'd run across issues in handling Unicode content in a recent awk (gawk) project. This is excellent news.
- sargstuff 4y agoSo, has anyone seen/awk'd a unicode mul-tics TeX check box example yet?
- sargstuff 4y agoPer discussion, header should be revised to "Awk kerning again a unicode mul-tics Tex thing"
- sytelus 4y agoBrian Kernighan is 80 years old. Donald Knuth is 84 years. Both still write code and avoid most social/non-social media in general. It's amazing given how often we get surrounded by people giving up coding at rather young age for various reasons. Who else would you include in this class?
- tialaramex 4y agoThe choice to use UTF-32 (ie Unicode code points as integers, which might as well be 32-bit since your CPU definitely doesn't have a suitably sized integer type) is unexpected, as I had seen so many other systems just choose to work entirely in UTF-8 for this problem. Now, Brian obviously has much better instincts about performance than I do and may even have tried some things and benchmarked them, but my guess would have been that you should stay in UTF-8 because it's always faster for the typical cases.
- bombcar 4y agoIs UTF-32 fixed size per char? Because then it allows simple math that you can’t do on UTF-8.
- moomin 4y agoA “character” can be of fairly arbitrary length in Unicode, so no.
- fooster 4y agoNot to be contradictory, but unicode is not a specific encoding. ufc-8 is an encoding (with a non specific length) and utf-32 is an encoding of a Unicode code point with a specific length.
- happytoexplain 4y agoMost people talking about characters in Unicode are referring to grapheme clusters, and I assume the parent is too, since they are in fact defined by Unicode and vary in length in all encodings, hence their answer of "no" to the GP.
- valleyer 4y agoIt's a fixed size per codepoint. Many clusters that appear atomic in a text editor are made up of multiple codepoints. The flag emojis are among the many examples.
- ducktective 4y agoI became aware of this while watching Professor Brailsford's interview with him (Computerphile channel): https://www.youtube.com/watch?v=GNyQxXw_oMQ https://www.youtube.com/watch?v=GNyQxXw_oMQ (Around 7-8 minute mark) Update: At 24-25 minute mark, he talks about the technologies he is inquiring to write his new book with (he mentions troff and groff). He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!" So there we go...We had the opportunity to read the mind of the developer of awk and unix and co-author of the literal "C Programming Language", confronting with the absolute state of the tooling of the modern world.
- dredmorbius 4y agoI just wanted to say: that's an excellent interview and I strongly recommend others listen to it.
- ChrisRR 4y ago> He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!" I'm struggling to understand this sentence. Who are they? What's a boonie? What's not finished?
- detaro 4y ago"in the boonies" = "in the wilderness/far away from everything/...". I.e. they probably have only crappy mobile internet and don't want to download large packages.
- mlyle 4y ago> He says he wanted to try "XeTeX" (which supports Unicode) but "...I was going to download it as an experiment and they wanted 5 gigabytes and 5 gigabytes at the particular boonies place I'm living would...mmm..not be finished yet!" Man, I think once you're Kernighan there should be like a 1gigabit/sec symmetric circuit wherever you go just in case you use it to do something else useful.
- svnpenn 4y agoLooks like its not really done yet: https://github.com/onetrueawk/awk/compare/master...unicode-support https://github.com/onetrueawk/awk/compare/master...unicode-s...
- etaioinshrdlu 4y agoI liked this interview with Brian with Lex Fridman: https://www.youtube.com/watch?v=O9upVbGSBFo https://www.youtube.com/watch?v=O9upVbGSBFo
- neilpanchal 4y ago+1. Also, I like your username.
- timakro 4y agoI believe no distro actually ships this version of awk by default. They ship GNU awk which has Unicode support anyways.
- svnpenn 4y agoDebian: https://distrowatch.com/table.php?distribution=debian&pkglist=true&version=11 https://distrowatch.com/table.php?distribution=debian&pkglis...
- timakro 4y agoSo it turns out the default on Debian is mawk which does NOT support Unicode. Thanks for pointing that out. This simple test gives different results for gawk and mawk. $ echo 'ö' | awk '{print length}'
- layer8 4y ago…only if the current locale is set to use UTF-8 (or some other variable-width encoding). Which nowadays the default locale usually does, but in principle it doesn’t need to be.
- chasil 4y agoOpenBSD uses "The One True AWK." $ awk -V awk version 20211208 Kernighan's version is likely used in other places where the GPL is eschewed.
- deleted 4y ago[deleted]
- cyocum 4y agoHere is Brian Kernighan mentioning the Unicode work in an interview: https://www.youtube.com/watch?v=GNyQxXw_oMQ https://www.youtube.com/watch?v=GNyQxXw_oMQ
- nanna 4y agoApparently Kernighan is also updating his Awk book of 1988 this summer too. https://irreal.org/blog/?p=10746 https://irreal.org/blog/?p=10746
- msravi 4y agoI've found this to be THE best book for learning awk and regular expressions. Its astounding how clear and structured the material is and how easy it is to retain in your head when presented that way.
- 7thaccount 4y agoThat would be cool
- wpietri 4y agoAs I get old to start thinking about exactly how long I want my career to be, this is inspiring. He is coming up on 80 and he's still coding.
- paledot 4y agoIs it? I'm glad it's an option, but I'm also glad retiring at 40 is an option (depending on a lot of stars aligning just right of course). My answer to "how long do I want my career to be?" will never be "until my body or mind fail me".
- deleted 4y ago[deleted]
- wpietri 4y agoDepends on how you think of it. Maybe I shouldn't have used the word "career" here, as I wrote code long before people paid me for it, and aim to keep doing it long after.
- jope12 4y agoDepends on what you mean with 'career'. Brian Kernighan doesn't do this because he needs money, but because he would get bored otherwise.
- arduinomancer 4y agoFor context this is 37 years after it was released (1985)
- deleted 4y ago[deleted]
- lelandfe 4y ago> Once I figure out how… I will try to submit a pull request. I wish I understood git better, but in spite of your help, I still don't have a proper understanding, so this may take a while. Even Kernighan struggles with git.
- Blikkentrekker 4y agoThe culture around p.r.s is truly a high barrier of entry for many people. Figuring out how all of this works is substantially more difficult I find in practice than fixing many longstanding trivial bugs in a great deal of software.
- School-Cotton 4y agoWhat’s the alternative? The old way (which is still used by many projects) is to send patches to mailing lists, which I find more difficult: you need to learn how to generate the patch from your source code repo, send the patch as an e-mail (needing weird hacks like `git imap-send`), and then configure your MUA not to mangle it somehow. Then you also don’t have a centralized search/tracking interface. Some good reasons not to use GitHub is because you’re familiar with standard/traditional tools, or because you prefer not to use centralized services. Both of those are fine reasons! But “the traditional way is easier” isn’t.
- Blikkentrekker 4y agoOne would no even need to make a diff in theory, simply the new version. In many cases, it is sufficient to simply send an email with: > There's a bug in the function `handle_request` in `src/network/core.rs` that can cause it to return a double answer when invoked during a leap second. This fixes that issue: ... And just include the new function inside of the email body, without attachment. That should be sufficient for many cases, but many projects do not allow this and demand p.r.s.
- wpietri 4y agoSeems like you're arguing against something they didn't say. They just said, "The culture around p.r.s is truly a high barrier of entry for many people." That's true and it's important to acknowledge that on any project that wants contributions from as many people as possible. When I was at Code for America I saw smart people putting plenty of time into helping people get over that barrier. That's certainly the case here. If a professor of CS receives help from an open-source maintainer who's been coding for decades and still doesn't feel confident, then it's safe to say a) there's room for improvement, and b) until we figure out what the improvement is, we should make sure that we're providing the necessary support to everybody we want to contribute.
- ducktective 4y ago[off-topic] Following the spirit of UNIX, I did a little analysis on the upvotes this post got over time (fish-shell): while true; curl -sL 'https://news.ycombinator.com/item?id=32534173' | pup '#score_32534173 text{}' | awk -F'[^0-9]*' '{print $1}' | tee -a points; sleep 15s; end (Initially I used `grep -Po '\d+'` but switched it with an awk solution due to...context!) I started it approx. when I posted it. Now ~2 hours have passed since. Using `gnuplot`: f(x) = a*x+b; fit f(x) "points" via a,b; set terminal png size 1920,1080 enhanced font "Inconsolata,20" ;set output "HN-analysis.png" ;set grid; set ylabel "points";set key bottom right ; set xlabel "sample # (15s interval)"; plot 'points' w linesp lt 7 lw 3 lc rgb "orange", f(x) lc rgb 'blue' lw 2 We generate the plot: https://i.imgur.com/pS6AaI5.png https://i.imgur.com/pS6AaI5.png (The jump #100 sample is due to a network error on my side) And here are the coeffs. of a linear fit over the data (note that every 4 samples is 1 minute, so this post got ~1.52 upvotes per minute) a = 0.380809, b = 19.8437
- sargstuff 4y ago?? nroff grap not installed ?? ;) https://n-t-roff.github.io/heirloom/doctools/grap.1.html https://n-t-roff.github.io/heirloom/doctools/grap.1.html
- ducktective 4y agoI didn't get it :) Are you suggesting another method for graphing?
- sargstuff 4y agoGRAP was/is Bell Labs developed software (by L. Bentley & Brian W. Kernighan ). gnuplot was/is Berkley / GNU developed software. Guess more of a 70's/80's pre-open source observational comment.
- sargstuff 4y agoGetting a t-shirt with "awk < 1" printed on it hand auto-grap-ed by Brian Kernighan would be more memorable though.
- stakkur 4y agoIt's somewhat comforting to hear even Brian K. say he doesn't understand Git well.
- seabrookmx 4y agoAnd to be fair he's been a professor for years, not a developer. When I took my undergrad roughly a decade ago, github was already a thing but our courses were still using subversion. Academia is slow to adopt new tools.
- layer8 4y agoI came to the comments to say that it’s reassuring. :)
- cafard 4y agoLikewise.
- rustqt6 4y agoGit really is a mess. The fact that commits and not diffs have hashes should be lampooned despite arguably a few small benefits. Geniuses make mistakes too and git is linuses. The only reason git is respected is because it came from Linus. If it were from Microsoft it would get all the criticism it deserves and then 20 times more
- nighthawk454 4y agoI never quite get this. Why would you want diffs to be the primary artifact? I’ve always found git to be exceedingly simple in design. Though, the CLI is a quite messy I grant you.
- benreesman 4y agoI think it would be a little more accurate to say that git is kind of a weird default: it’s a revision control power-user tool with all the sharp corners we expect from such tools and is overkill for most users in most situations. It’s understandable that typical users resent the complexity and the foot cannons. Whether it’s git or vi/emacs or C or LaTex or whatever, there are a bunch of us old-timers who went through the pain already and now notice the modest loss of capability on hg or VSCode or whatever. But even as such an old-timer, I don’t think it’s a good idea to default everyone into this low-level shit. Mercurial and VSCode and Python are fine, you can do great work without “the bare metal”. Edit: Clarified that it’s a “revision control” power-user tool, plenty of “power users” have more typical revision-control needs.
- pid_0 4y ago
- YesThatTom2 4y agoOf course he did. Aho has better things to do and Weinberger is too rich to write code any more.
- deleted 4y ago[deleted]
- bear8642 4y agoWhilst great Brian's added this, didn't Plan9's version of AWK support UTF-8? I haven't used it, but my understanding was all the unix tools were upgraded to work with unicode - don't quite see why it's taken so long if the work was done already years ago