12 ms·

Unicode is harder than you think

126 points by mcilloni 3y ago

night-rider 3y ago
It's not that it's hard, it's just people don't go out of their way to escape UTF-8 glyphs into ASCII when dealing with exotic glyphs in a text editor. It's more mundane and tedious, but not 'hard'. Try working with raw UTF-8 in JS and find yourself in a world of pain. Mathias Bynens talks about these gotchas here: https://mathiasbynens.be/notes/javascript-unicode https://mathiasbynens.be/notes/javascript-unicode
- JohnFen 3y ago
  "Hard" is a bit context-dependent. Instead of thinking of it as "hard", I think of it as a real pain in the ass full of footguns.
dmitrygr 3y ago
Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags https://esham.io/2014/06/unicode-flags
- svachalek 3y ago
  Interesting. They pushed all the politics onto the font designers.
  - not2b 3y ago
    The font designer has to include a flag for each supported country. Often a given font is missing lots of flags for reasons that have nothing to do with whether the designer recognizes a given country or not, just a question of priorities; perhaps only 100 out of 200 flags are supported.
- Longhanks 3y ago
  Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.
  - qalmakka 3y ago
    If it were that easy - sadly everything that's in any way related to the way we communicate and we relate to the world. Just look at the kerfuffle about skin tones... Everything is political if you are looking for a reason to fight. Language is a very sensitive topic - in Central Asian countries, using Latin, Cyrillic or Perso-Arabic script for instance has very strong political connotations, same in the Balkans. The world is just like that
  - veave 3y ago
    Or when big tech banded together to change the pistol emoji to some scifi zapper.
    - spookthesunset 3y ago
      From what I recall the problem was on some devices it was rendered as a “sci-fi zapper” or squirtgun and on others it was a fairly realistic depiction of a gun. Leading to some misunderstandings…
    - throwawaymobule 3y ago
      I think that was a case of apple leading and everyone else following. They went with a water pistol in their font.
  - makeworld 3y ago
    There is no way to avoid it. It is very obvious that deciding whether "vaccine syringes" are political (and therefore excluded) or not is itself a political decision.
    - naniwaduni 3y ago
      There's a certain kind of extremist who claims that their contentious positions aren't political, but the fact that there's an argument, and that you can point to mainstream coverage of it, strongly suggests that they're full of shit and everyone can see it.
- rjh29 3y ago
  This is actually a genius decision on their part! It's telling that for Han unification they stomped all over Japanese/Korean/Vietnamese glyphs but still kept the simplified and traditional forms separate.
jmclnx 3y ago
No kidding, you have not lived until you try and explain UTF-8 to people who only believes in what they called "doublebyte". You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.
- theamk 3y ago
  Thank god for emojis! Those people would say, "No one in our org would use chinese" and refuse to fix things... but now I just point them to latest message from upper management which contain emoji or two. (And emoji are such a fine example - once they ate on the table, you need support for combining characters, characters outside of BMP, ligatures.. a large part of Unicode spec)
  - throwawaymobule 3y ago
    It's a very strange series of events that led us here, with various stages of 'oh no' and 'oh yes' events happening. Like Apple trying to region-lock emojis, or whoever decided merging CJK together was a good idea, or that plan9 dev who scribbled out UTF8 on a napkin.
- qalmakka 3y ago
  It's terrible, and we IMHO owe that to some introductory university courses to Java (plus some Win32 veterans). I got very close to being rejected by a professor that was obstinately convinced that Unicode "characters" were 2 bytes because it drunk the Kool Aid in the '90s about Java's `char` type representing a Unicode character. Ugh. I still get angry by thinking back at that sometimes
  - jmclnx 3y ago
    I can relate, I remember a teacher stating "you never have to worry about the amount of memory". This was in the late 90s, I then asked "So I can load a 20 gig data file into memory", he said yes.
spudlyo 3y ago
If you found this essay interesting, you owe it to yourself to check out this super entertaining talk "Plain Text"[0] from NDC 2022 by Dylan Beattie. Rabbit hole warning: This video caused me to lose an entire Sunday watching Dylan's talks on YouTube, which are uniformly awesome. [0]: https://www.youtube.com/watch?v=gd5uJ7Nlvvo https://www.youtube.com/watch?v=gd5uJ7Nlvvo
- WaffleIronMaker 3y ago
  I also really enjoy Dylan Beattie's work. For those with some spare time, who might like to see a true "rockstar" programmer, you may like "The Art of Code"[0]. [0] https://youtu.be/6avJHaC3C2U https://youtu.be/6avJHaC3C2U
  - spudlyo 3y ago
    Another good talk! I also really loved "Email vs Capitalism, or, Why We Can't Have Nice Things"[0] which has one of the best audience participation gimmicks I have ever seen in a talk. [0]: https://www.youtube.com/watch?v=mrGfahzt-4Q https://www.youtube.com/watch?v=mrGfahzt-4Q
  - aidos 3y ago
    Oh wow. The Amstrad 6128 was my first machine (1985). Looking forward to watching this!
- soneil 3y ago
  This is exactly what I came here to post, too. I believe he's unusually prolific because he's part of the organisation of these conferences - but his delivery pays off regardless. I actually sent exactly the same video to a colleague this a few days ago.
zamadatix 3y ago
TIL of UTF-1, what an odd specification.
- paulddraper 3y ago
  Yeah, what an odd spec. [1] Looks like the entire idea is to avoid using any ASCII/Extended ASCII control character bytes, except to represent those characters. Not sure what the advantage of that was. [1] https://en.wikipedia.org/wiki/UTF-1 https://en.wikipedia.org/wiki/UTF-1
jkaptur 3y ago
The logical next step here is to realize that if you want to be truly internationalized, pretty much every single method of the string class in your favorite language is an antipattern and should be used with extreme caution. Seriously!
- frizlab 3y ago
  I _think_ Swift did it properly. At least that’s what they claim[1] [1]: https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/ There were many discussions on how to handle strings in the forums too. Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly. All in all most beginners in Swift hate working with strings because it’s not intuitive at first, but to be fair in almost all languages strings handling is not done properly.
  - jkaptur 3y ago
    Oh, interesting! The fact that "abc"[1] doesn't work is a great sign, since my contention is that "the character at index i" is not a well-defined concept.
Pannoniae 3y ago
Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII) (ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)
- david_draco 3y ago
  Maybe an "Acid test" for Unicode would help? These pages seem to go into that direction: https://www.kermitproject.org/utf8.html https://www.kermitproject.org/utf8.html https://web.archive.org/web/20160306060703/http://www.inter-locale.com/whitepaper/learn/learn-to-test.html https://web.archive.org/web/20160306060703/http://www.inter-... Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.
  - techdragon 3y ago
    I was tempted to include unicode when I enhanced the URL/Domain generating strategies in Hypothesis to correctly generate from the set of all valid TLDs a few years ago! But I decided against it since technically the canonical storage form doesn't contain the unicode characters as it should have had Punycode (https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode ) conversion done by the "input" system be it a URL bar of your browser or whatever input handling is in play. Assuming all domain processors are able to correctly Punycode was a bit too much, so I kept it to the set of basic valid characters without needing Punycode conversion, and if someone wanted to fuzz test fully they could just add a layer that generates from full unicode.
- 3y ago
NovemberWhiskey 3y ago
I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strings were null-terminated byte streams. The platform had CSV import capability for tabular data, and it had an integrated pivot table capability based on some widgets that had been grafted onto it. Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this. My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.
- GuB-42 3y ago
  UTF-8 is made in such a way that processing based on ASCII and bytes mostly works. All non-ASCII characters are just a sequence of bytes >127, and the system can recover if code points are split or truncated. So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.
nightpool 3y ago
Thank you for being the first article I've ever actually read to explain the difference between NFC, NFD, NFKD and NFKC in a way that I actually understood. I was a little bored through the whole UCS/UTF* history lesson because I knew a lot of it already, but the normalization and collation examples were definitely worth it
- Lammy 3y ago
  Agreed, and it would be even better if it mentioned some real-world normalization issues like it does for the UCS encodings. I learned about it the hard way when dealing with Apple filesystems: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/ https://eclecticlight.co/2021/05/08/explainer-unicode-normal...
  - seoulbigchris 3y ago
    Man, this gave me such a big headache when using a NAS with MacOS. I have a bunch of file names that use Hangul, and it resulted in a mess that I never fully sorted out.
alcover 3y ago
Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?
- ekidd 3y ago
  One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points). This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type. For another good alternative to this approach, see Ruby.
  - Animats 3y ago
    > separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Yes. Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as s(5)Hello This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems. Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check. Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as (SMILING FACE)(RED HEART) per the spec.
o1y32 3y ago
Regarding the title -- anecdotally, everyone I know is sacred of encoding issues, and I don't know anyone who claims they have a great understanding of Unicode or think it is easy (including myself). It is often overlooked for sure -- people don't realize there is a problem in the code until they run into a bug, ane it turns out they are treating strings wrong from the very beginning.
- m_0x 3y ago
  sacred or scared? ;)
skitter 3y ago
Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.
- deadbeeves 3y ago
  But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?
  - skitter 3y ago
    Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8 https://simonsapin.github.io/wtf-8/#utf-8
    - deadbeeves 3y ago
      It doesn't sound very annoying, then. You use the exact same encoding scheme, but skip a verification step. Actually it sounds more convenient.
      - jraph 3y ago
        Still potentially annoying if you deal with some other code that expects UTF-8 proper and you pass it a wtf-8 string that fails the lifted verification.
  - deleted 3y ago
    [deleted]
- sedatk 3y ago
  Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surrogates-and-supplementary-characters https://learn.microsoft.com/en-us/windows/win32/intl/surroga...
Roark66 3y ago
Indeed it is. One use of Unicode I do is for icons that can be used by console programs like (neo)vim. I was quite happy that xterm supports Unicode these days so I can use a fast terminal that supports OSC52 system clipboard integration(none of the newer gnome/KDE terminals do). I was rather disappointed when I noticed my pretty Unicode icons would sometimes end up cut in half :-(
ggm 3y ago
if you see URLs which project the microsoft editors choice of "single quote mark" into a triad of %xx; values, you've seen how people wind up in unexpected unicode/utf land, because of the text editor swapping ASCII for an uplift into either distorted iso-latin1, or unicode/utf8 depending.
- Calzifer 3y ago
  Annoying. Had recently trouble with a translation file (Java properties format). Traditionally it is all ASCII with \u1234 escape sequences. Well except for that recently added "single quote" which is actually some unnecessary (and I assume unwanted) more exotic Unicode character. Also a reason why I'm against Unicode identifier in source code. You got some source with characters like 'ö' in one variant and now your editor decides to use a different variant (e.g. compatibility character vs base character + combining decoration). And from my observation programming languages which claim to support Unicode tend to not make the effort to resolve those equivalents. At least for older versions of Go the specification clearly stated that they only compare Unicode code points. (don't know the current state)
Amorymeltzer 3y ago
I'll always take an excuse to link to one of my favorite StackOverflow answers, to the question "Why does modern Perl avoid UTF-8 by default?": <https://stackoverflow.com/a/6163129/2521092 https://stackoverflow.com/a/6163129/2521092>. It's from 2011 and Perl-centric, of course, but skip down to "𝔸 𝕤 𝕤 𝕦 𝕞 𝕖 𝔹 𝕣 𝕠 𝕜 𝕖 𝕟 𝕟 𝕖 𝕤 𝕤" for a thorough, if opinionated, list. My favorite, similar to the issue of case-folding, is the idea of something being a "letter" and "lowercase" but NOT "lowercase letter."
- rjh29 3y ago
  I loved that answer too but tchrist didn't explain why any of the assumptions were wrong! That led me down a rabbit hole and I did it myself: https://richardjharris.github.io/all-sorts-of-things-you-can-get-wrong-in-unicode-and-why/ https://richardjharris.github.io/all-sorts-of-things-you-can...
solresol 3y ago
The article didn't mention UTF-9 or UTF-18 for when you want to do unicode on your PDP-6 or Univac. https://datatracker.ietf.org/doc/html/rfc4042 https://datatracker.ietf.org/doc/html/rfc4042 A reviewer for a scientific paper I wrote asked if I had considered the impact of UTF-9 on what I was doing. To this day I don't know if they were trolling me.
butlerm 3y ago
With the benefit of hindsight, if the standard was done over again either a lot of unnecessary difficulties mentioned in this article should be avoided or the standard split in two. It is arguably too ambitious, too inefficient, and too unrealistic for a number of these things to be handled as recommended in all contexts. There are many examples - in operating system kernels and system programming in general, to name two. To start with, alternative representations for the same visibly identical character should have been excluded. In the base standard all supported characters should be precomposed - no modifiers. The article points out the difficulties in something as simple (and as security critical) as determining whether two strings refer to the same visibly identical characters. A character set that does not make that trivial is not suitable for general use in system programming or for security critical identifiers (unfortunately). It massively complicates programming in many programming languages as well, and even UTF-32 is not sufficient to remedy the problem, as the article well notes. The inability to handle and process any arbitrary string of bytes in a universal character set is a serious problem as well. The world is complicated, and the inability to pass through incorrectly coded data, alternatively coded data, and general binary data is a major limitation with serious consequences. System code such as device drivers or filesystems typically can't deal with inefficiencies or limitations like that. In addition, there probably should be two types of uppercase and lowercase conversions, a simple, predictable one for system programming, and a more complex one that deals with considerations in languages that do not follow the normal rules. String collation should be done at two levels as well, a simple code point level suitable for such things as prefix matches on database indexes, and a more complex variation for applications where linguistically sensitive collation is critical. In general trying to solve all the higher end use cases in a large body of software that do not need to deal with them, should not need to deal with them, or cannot deal with them is impractical and has exacerbated the common string processing issues we see today. A lightweight standard - perhaps a subset or profile of Unicode that supported arbitrary binary data as opaque codepoints - could be helpful in a lot of contexts.
eviks 3y ago
Pity those bad design decisions like surrogate pair pollute the future for generations instead of being cleanly cut off with a painful, but temporary transition period
_nalply 3y ago
Another important subject are confusables. "A" looks same for Latin A, Greek Alpha and Cyrillic A. If you need users to identify names by sight, you need to do something about confusables. It is even a security problem, for example for hostnames, imagine a hacker build your bank's website with a similar looking URL. https://news.ycombinator.com/item?id=32497414 https://news.ycombinator.com/item?id=32497414
qingcharles 3y ago
What is most interesting to me is that some countries just give up completely on their native script: https://en.wikipedia.org/wiki/Spread_of_the_Latin_script#21st_century https://en.wikipedia.org/wiki/Spread_of_the_Latin_script#21s... Any Indians here? I heard that most Hindi texting is done with a weird Latin character slang these days?

126 points by mcilloni 3y ago

It's not that it's hard, it's just people don't go out of their way to escape UTF-8 glyphs into ASCII when dealing with exotic glyphs in a text editor. It's more mundane and tedious, but not 'hard'. Try working with raw UTF-8 in JS and find yourself in a world of pain. Mathias Bynens talks about these gotchas here: https://mathiasbynens.be/notes/javascript-unicode https://mathiasbynens.be/notes/javascript-unicode

JohnFen 3y ago
"Hard" is a bit context-dependent. Instead of thinking of it as "hard", I think of it as a real pain in the ass full of footguns.

dmitrygr 3y ago

Favourite unicode fact: properly rendering unicode requires understanding of the current geopolitical situation (Depending on whom you accept as a country and whom you do not, two country-code-letters may or may not render as a flag. This changes sometimes in today's world.). https://esham.io/2014/06/unicode-flags https://esham.io/2014/06/unicode-flags

svachalek 3y ago
Interesting. They pushed all the politics onto the font designers.
- not2b 3y ago
  The font designer has to include a flag for each supported country. Often a given font is missing lots of flags for reasons that have nothing to do with whether the designer recognizes a given country or not, just a question of priorities; perhaps only 100 out of 200 flags are supported.
Longhanks 3y ago
Imho, unicode should stay out of politics. Country flags, vaccine syringes and pregnant men should have nothing to do with how computers handle text and writing systems.
- qalmakka 3y ago
  If it were that easy - sadly everything that's in any way related to the way we communicate and we relate to the world. Just look at the kerfuffle about skin tones... Everything is political if you are looking for a reason to fight. Language is a very sensitive topic - in Central Asian countries, using Latin, Cyrillic or Perso-Arabic script for instance has very strong political connotations, same in the Balkans. The world is just like that
- veave 3y ago
  Or when big tech banded together to change the pistol emoji to some scifi zapper.
  - spookthesunset 3y ago
    From what I recall the problem was on some devices it was rendered as a “sci-fi zapper” or squirtgun and on others it was a fairly realistic depiction of a gun. Leading to some misunderstandings…
  - throwawaymobule 3y ago
    I think that was a case of apple leading and everyone else following. They went with a water pistol in their font.
- makeworld 3y ago
  There is no way to avoid it. It is very obvious that deciding whether "vaccine syringes" are political (and therefore excluded) or not is itself a political decision.
  - naniwaduni 3y ago
    There's a certain kind of extremist who claims that their contentious positions aren't political, but the fact that there's an argument, and that you can point to mainstream coverage of it, strongly suggests that they're full of shit and everyone can see it.
rjh29 3y ago
This is actually a genius decision on their part! It's telling that for Han unification they stomped all over Japanese/Korean/Vietnamese glyphs but still kept the simplified and traditional forms separate.

jmclnx 3y ago

No kidding, you have not lived until you try and explain UTF-8 to people who only believes in what they called "doublebyte". You think they get it, but surprise happens when a database load fails when loading Chinese Character "string" into a field sized calculated based upon 2 bytes per character.

theamk 3y ago
Thank god for emojis! Those people would say, "No one in our org would use chinese" and refuse to fix things... but now I just point them to latest message from upper management which contain emoji or two. (And emoji are such a fine example - once they ate on the table, you need support for combining characters, characters outside of BMP, ligatures.. a large part of Unicode spec)
- throwawaymobule 3y ago
  It's a very strange series of events that led us here, with various stages of 'oh no' and 'oh yes' events happening. Like Apple trying to region-lock emojis, or whoever decided merging CJK together was a good idea, or that plan9 dev who scribbled out UTF8 on a napkin.
qalmakka 3y ago
It's terrible, and we IMHO owe that to some introductory university courses to Java (plus some Win32 veterans). I got very close to being rejected by a professor that was obstinately convinced that Unicode "characters" were 2 bytes because it drunk the Kool Aid in the '90s about Java's `char` type representing a Unicode character. Ugh. I still get angry by thinking back at that sometimes
- jmclnx 3y ago
  I can relate, I remember a teacher stating "you never have to worry about the amount of memory". This was in the late 90s, I then asked "So I can load a 20 gig data file into memory", he said yes.

spudlyo 3y ago

If you found this essay interesting, you owe it to yourself to check out this super entertaining talk "Plain Text"[0] from NDC 2022 by Dylan Beattie. Rabbit hole warning: This video caused me to lose an entire Sunday watching Dylan's talks on YouTube, which are uniformly awesome. [0]: https://www.youtube.com/watch?v=gd5uJ7Nlvvo https://www.youtube.com/watch?v=gd5uJ7Nlvvo

WaffleIronMaker 3y ago
I also really enjoy Dylan Beattie's work. For those with some spare time, who might like to see a true "rockstar" programmer, you may like "The Art of Code"[0]. [0] https://youtu.be/6avJHaC3C2U https://youtu.be/6avJHaC3C2U
- spudlyo 3y ago
  Another good talk! I also really loved "Email vs Capitalism, or, Why We Can't Have Nice Things"[0] which has one of the best audience participation gimmicks I have ever seen in a talk. [0]: https://www.youtube.com/watch?v=mrGfahzt-4Q https://www.youtube.com/watch?v=mrGfahzt-4Q
- aidos 3y ago
  Oh wow. The Amstrad 6128 was my first machine (1985). Looking forward to watching this!
soneil 3y ago
This is exactly what I came here to post, too. I believe he's unusually prolific because he's part of the organisation of these conferences - but his delivery pays off regardless. I actually sent exactly the same video to a colleague this a few days ago.

zamadatix 3y ago

TIL of UTF-1, what an odd specification.

paulddraper 3y ago
Yeah, what an odd spec. [1] Looks like the entire idea is to avoid using any ASCII/Extended ASCII control character bytes, except to represent those characters. Not sure what the advantage of that was. [1] https://en.wikipedia.org/wiki/UTF-1 https://en.wikipedia.org/wiki/UTF-1

jkaptur 3y ago

The logical next step here is to realize that if you want to be truly internationalized, pretty much every single method of the string class in your favorite language is an antipattern and should be used with extreme caution. Seriously!

frizlab 3y ago
I _think_ Swift did it properly. At least that’s what they claim[1] [1]: https://www.swift.org/blog/utf8-string/ https://www.swift.org/blog/utf8-string/ There were many discussions on how to handle strings in the forums too. Remarkably, it is not possible to access "abc"[1] as it has unpredictable performance; instead one has to build the index and making this should make the developer realize the operation is costly. All in all most beginners in Swift hate working with strings because it’s not intuitive at first, but to be fair in almost all languages strings handling is not done properly.
- jkaptur 3y ago
  Oh, interesting! The fact that "abc"[1] doesn't work is a great sign, since my contention is that "the character at index i" is not a well-defined concept.

Pannoniae 3y ago

Most programs claim to support Unicode but they actually don't. They either miscount string lengths (you type a CJK character or an emoji in, string appears shorter than what the program thinks), separate them improperly or many other things. It doesn't help that by default, most programming languages also handle unicode poorly, with the default APIs producing wrong results. I'd take "we don't do unicode at all" or "we only support BMP" or "we don't support composite characters" any day over pretend-support (but then inevitably breaking when the program wasn't tested with anything non-ASCII) (ninjaedit: to see how prevalent it is, even gigantic message apps such as discord make this mistake. There are users on discord who you can't add as friends because the friend input field is limited to 32.... something - probably bytes, yet elsewhere the program allows the name to be taken. This is easy to do with combining characters)

david_draco 3y ago
Maybe an "Acid test" for Unicode would help? These pages seem to go into that direction: https://www.kermitproject.org/utf8.html https://www.kermitproject.org/utf8.html https://web.archive.org/web/20160306060703/http://www.inter-locale.com/whitepaper/learn/learn-to-test.html https://web.archive.org/web/20160306060703/http://www.inter-... Placing a fuzzy tester like "hypothesis.strategies.characters" into the CI may also be revealing.
- techdragon 3y ago
  I was tempted to include unicode when I enhanced the URL/Domain generating strategies in Hypothesis to correctly generate from the set of all valid TLDs a few years ago! But I decided against it since technically the canonical storage form doesn't contain the unicode characters as it should have had Punycode (https://en.wikipedia.org/wiki/Punycode https://en.wikipedia.org/wiki/Punycode ) conversion done by the "input" system be it a URL bar of your browser or whatever input handling is in play. Assuming all domain processors are able to correctly Punycode was a bit too much, so I kept it to the set of basic valid characters without needing Punycode conversion, and if someone wanted to fuzz test fully they could just add a layer that generates from full unicode.
3y ago

NovemberWhiskey 3y ago

I used to work on a platform at a large financial services firm; it was essentially complete ignorant of anything Unicode with respect to string handling, strings were null-terminated byte streams. The platform had CSV import capability for tabular data, and it had an integrated pivot table capability based on some widgets that had been grafted onto it. Some of the users in Hong Kong discovered that you could import CSVs with Unicode text (e.g. index compositions with Chinese company names) and they'd display in the pivot table widgets and even be exportable to reports. But only most names. Some names were truncated or turned into garbage, and I was called upon to help debug this. My first reaction was frank amazement that this "worked" at all: apparently, the path from the dumb CSV import code through to the Unicode-aware pivot table was sufficiently clean that much of the encoded text made it through OK. I can't remember the precise details now but I think the problem turned out to be embedded nulls from UTF-16 encoding and so was completely insoluble without a gut renovation of the platform.

GuB-42 3y ago
UTF-8 is made in such a way that processing based on ASCII and bytes mostly works. All non-ASCII characters are just a sequence of bytes >127, and the system can recover if code points are split or truncated. So if you just move strings around and concatenate them, you don't even need to be aware of Unicode, even things like templating can work by just processing bytes. Of course, when you get to editing and rendering, you get all the complexity.

nightpool 3y ago

Thank you for being the first article I've ever actually read to explain the difference between NFC, NFD, NFKD and NFKC in a way that I actually understood. I was a little bored through the whole UCS/UTF* history lesson because I knew a lot of it already, but the normalization and collation examples were definitely worth it

Lammy 3y ago
Agreed, and it would be even better if it mentioned some real-world normalization issues like it does for the UCS encodings. I learned about it the hard way when dealing with Apple filesystems: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/ https://eclecticlight.co/2021/05/08/explainer-unicode-normal...
- seoulbigchris 3y ago
  Man, this gave me such a big headache when using a NAS with MacOS. I have a bunch of file names that use Hangul, and it resulted in a mess that I never fully sorted out.

alcover 3y ago

Currently working on a language, I feel dizzy after reading this. My stdlib will provide a (byte) Buffer class with basic low-level methods but I feel like iterating through it in fancy ways should be the concern of the user or 3rd-party libraries. I fail to see this as part of a programming language. Am I wrong here ?

ekidd 3y ago
One good approach is to have separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Then you can have a nice, user-friendly string class for basic UTF-8 text, which is pretty easy. Ignore sorting and grapheme clusters (those probably belong in libraries, and they require fairly large tables). Consider providing a library function to iterate over UTF-8 "characters" (as unpacked 32 bit Unicode code points). This is one of the sweet spots in language design, and it provides enough structure for third-party libraries to work well together, without everyone reinventing their own string type. For another good alternative to this approach, see Ruby.
- Animats 3y ago
  > separate "byte array" and "string" types, and say, "Strings are always UTF-8. Anything else is a bug. Deal with it." Yes. Current headache: I have spent the last few days writing a parser for a data representation with far too much embedding. The innermost layer is JSON, representing glTF graphics data. That's UTF-8. That's encapsulated in an obsolete format called "Notation", which is sort of a pre-JSON kind of JSON. It's a byte stream, and has byte counted forms such as s(5)Hello This, in turn, is encapsulated in UTF-8 XML. I was coding this in Rust, which led to the discovery that the C++ code it is emulating had problems. Rust has the right concept. Strings are UTF-8, strictly enforced, so you can iterate over them. Byte arrays are arrays of 8-bit unsigned integers. Conversions check. Classic headache: turning SMS texts into ITA-2 for an antique Teletype machine for steampunk demos. Anything outside of the ITA-2 set (uppercase letters, numbers, and some punctuation) was turned into the long text description of that code point. Emoji text were dutifully hammered out as (SMILING FACE)(RED HEART) per the spec.

o1y32 3y ago

Regarding the title -- anecdotally, everyone I know is sacred of encoding issues, and I don't know anyone who claims they have a great understanding of Unicode or think it is easy (including myself). It is often overlooked for sure -- people don't realize there is a problem in the code until they run into a bug, ane it turns out they are treating strings wrong from the very beginning.

m_0x 3y ago
sacred or scared? ;)

skitter 3y ago

Annoyingly, Java, JavaScript, Windows file paths and more don't quite use UTF-16 (well, even if they did, that would be annoying) — they allow unpaired surrogates, which don't represent any Unicode character. So if you want to represent e.g. an arbitrary Windows file path in UTF-8, you can't; you have to use WTF-8 (wobbly transformation format) instead.

deadbeeves 3y ago
But UTF-8 is just a way to encode a number as a variable-length string of octets. Why would you be unable to encode, say, a terminating U+D800 as a string of three bytes at the end of a UTF-8 stream?
- skitter 3y ago
  Because that's how UTF-8 is defined[1]. WTF-8 lifts that restriction. [1] https://simonsapin.github.io/wtf-8/#utf-8 https://simonsapin.github.io/wtf-8/#utf-8
  - deadbeeves 3y ago
    It doesn't sound very annoying, then. You use the exact same encoding scheme, but skip a verification step. Actually it sounds more convenient.
    - jraph 3y ago
      Still potentially annoying if you deal with some other code that expects UTF-8 proper and you pass it a wtf-8 string that fails the lifted verification.
- deleted 3y ago
  [deleted]
sedatk 3y ago
Certainly not true for Windows. Windows uses UTF-16; e.g. it uses proper surrogate pairs. https://learn.microsoft.com/en-us/windows/win32/intl/surrogates-and-supplementary-characters https://learn.microsoft.com/en-us/windows/win32/intl/surroga...

Roark66 3y ago

Indeed it is. One use of Unicode I do is for icons that can be used by console programs like (neo)vim. I was quite happy that xterm supports Unicode these days so I can use a fast terminal that supports OSC52 system clipboard integration(none of the newer gnome/KDE terminals do). I was rather disappointed when I noticed my pretty Unicode icons would sometimes end up cut in half :-(

ggm 3y ago

if you see URLs which project the microsoft editors choice of "single quote mark" into a triad of %xx; values, you've seen how people wind up in unexpected unicode/utf land, because of the text editor swapping ASCII for an uplift into either distorted iso-latin1, or unicode/utf8 depending.

Calzifer 3y ago
Annoying. Had recently trouble with a translation file (Java properties format). Traditionally it is all ASCII with \u1234 escape sequences. Well except for that recently added "single quote" which is actually some unnecessary (and I assume unwanted) more exotic Unicode character. Also a reason why I'm against Unicode identifier in source code. You got some source with characters like 'ö' in one variant and now your editor decides to use a different variant (e.g. compatibility character vs base character + combining decoration). And from my observation programming languages which claim to support Unicode tend to not make the effort to resolve those equivalents. At least for older versions of Go the specification clearly stated that they only compare Unicode code points. (don't know the current state)

Amorymeltzer 3y ago

I'll always take an excuse to link to one of my favorite StackOverflow answers, to the question "Why does modern Perl avoid UTF-8 by default?": <https://stackoverflow.com/a/6163129/2521092 https://stackoverflow.com/a/6163129/2521092>. It's from 2011 and Perl-centric, of course, but skip down to "𝔸 𝕤 𝕤 𝕦 𝕞 𝕖 𝔹 𝕣 𝕠 𝕜 𝕖 𝕟 𝕟 𝕖 𝕤 𝕤" for a thorough, if opinionated, list. My favorite, similar to the issue of case-folding, is the idea of something being a "letter" and "lowercase" but NOT "lowercase letter."

rjh29 3y ago
I loved that answer too but tchrist didn't explain why any of the assumptions were wrong! That led me down a rabbit hole and I did it myself: https://richardjharris.github.io/all-sorts-of-things-you-can-get-wrong-in-unicode-and-why/ https://richardjharris.github.io/all-sorts-of-things-you-can...

solresol 3y ago

The article didn't mention UTF-9 or UTF-18 for when you want to do unicode on your PDP-6 or Univac. https://datatracker.ietf.org/doc/html/rfc4042 https://datatracker.ietf.org/doc/html/rfc4042 A reviewer for a scientific paper I wrote asked if I had considered the impact of UTF-9 on what I was doing. To this day I don't know if they were trolling me.

butlerm 3y ago

With the benefit of hindsight, if the standard was done over again either a lot of unnecessary difficulties mentioned in this article should be avoided or the standard split in two. It is arguably too ambitious, too inefficient, and too unrealistic for a number of these things to be handled as recommended in all contexts. There are many examples - in operating system kernels and system programming in general, to name two. To start with, alternative representations for the same visibly identical character should have been excluded. In the base standard all supported characters should be precomposed - no modifiers. The article points out the difficulties in something as simple (and as security critical) as determining whether two strings refer to the same visibly identical characters. A character set that does not make that trivial is not suitable for general use in system programming or for security critical identifiers (unfortunately). It massively complicates programming in many programming languages as well, and even UTF-32 is not sufficient to remedy the problem, as the article well notes. The inability to handle and process any arbitrary string of bytes in a universal character set is a serious problem as well. The world is complicated, and the inability to pass through incorrectly coded data, alternatively coded data, and general binary data is a major limitation with serious consequences. System code such as device drivers or filesystems typically can't deal with inefficiencies or limitations like that. In addition, there probably should be two types of uppercase and lowercase conversions, a simple, predictable one for system programming, and a more complex one that deals with considerations in languages that do not follow the normal rules. String collation should be done at two levels as well, a simple code point level suitable for such things as prefix matches on database indexes, and a more complex variation for applications where linguistically sensitive collation is critical. In general trying to solve all the higher end use cases in a large body of software that do not need to deal with them, should not need to deal with them, or cannot deal with them is impractical and has exacerbated the common string processing issues we see today. A lightweight standard - perhaps a subset or profile of Unicode that supported arbitrary binary data as opaque codepoints - could be helpful in a lot of contexts.

eviks 3y ago

Pity those bad design decisions like surrogate pair pollute the future for generations instead of being cleanly cut off with a painful, but temporary transition period

_nalply 3y ago

Another important subject are confusables. "A" looks same for Latin A, Greek Alpha and Cyrillic A. If you need users to identify names by sight, you need to do something about confusables. It is even a security problem, for example for hostnames, imagine a hacker build your bank's website with a similar looking URL. https://news.ycombinator.com/item?id=32497414 https://news.ycombinator.com/item?id=32497414

qingcharles 3y ago

What is most interesting to me is that some countries just give up completely on their native script: https://en.wikipedia.org/wiki/Spread_of_the_Latin_script#21st_century https://en.wikipedia.org/wiki/Spread_of_the_Latin_script#21s... Any Indians here? I heard that most Hindi texting is done with a weird Latin character slang these days?