9 ms·

I think we should take a moment to appreciate how great UTF-8 is, and how it well it worked out. It's easy to get disillusioned with internet standards when IPv

I think we should take a moment to appreciate how great UTF-8 is, and how it well it worked out. It's easy to get disillusioned with internet standards when IPv6 is taking forever and messaging is all proprietary locked down protocols. Yet character encodings used to be a horrible mess and now it's not. In the 90's the only practical solution was for everyone to use the same OS, same word processor, same web browser, and who cares about talking with foreigners anyway? I don't think it was always guaranteed to turn out well. China and Japan could have stayed with their own encodings. Microsoft and Apple could have done incompatible things. The tech world is full of bad things we're stuck with because there's no way to coordinate a change. Unicode has it's flaws, UTF-16 is still lurking here and there, everyone loves to argue about emoji, but overall text just works now.

roblabla 6y ago
> UTF-16 is still lurking here and there As someone currently stuck in the windows world, this hurts. Every single Windows API is still stuck with using UTF-16/UCS2 as the string encoding. Also fun fact, on the Nintendo Switch, various subsystems use different kind of encoding. The filsystem submodule uses Shift-JIS, most of the other modules use UTF-8, but some others yet use UTF-16 (like the virtual keyboard, IIRC). A brilliant mess.
- ChrisSD 6y ago
  Personally I don't find UTF-16 to be too bad. It's a simple encoding and very easy to convert to/from UTF-8. So your program can be written in UTF-8 and your WinAPI wrappers can convert as/when needed.
  - jeltz 6y ago
    The bad thing with UTF-16 is that so much software assumes that one code point always is 16 bits.
    - cygx 6y ago
      Most of the hard stuff is there no matter the encoding (normalization, user-perceived characters spanning multiple code units, paths vs strings, ...).
    - jrochkind1 6y ago
      Which is not UTF-16 at all, UTF-16 standard clearly says this is not so. So why do they do that? It's actually a leftover of the earlier UCS-2 standard, before it was realized we'd need more codepoints than that, and that it was a mistake to limit to 16-bit space for codepoints in any encoding. Software written for UCS-2 can mostly work compatibly with UTF-16, but there are some problems, encoding the 'higher' codepoints is only one of several. Another is how right-to-left scripts are handled. http://www.differencebetween.net/technology/software-technology/difference-between-ucs-2-and-utf-16/ http://www.differencebetween.net/technology/software-technol... https://unicode.org/faq/utf_bom.html#utf16-11 https://unicode.org/faq/utf_bom.html#utf16-11
      - flohofwoe 6y ago
        Wasn't UTF-16 explicitly created as a "backward compatibility hack" for UCS-2 when it became clear that 16 bits per code point isn't enough? They should have ditched 16-bit encodings back then instead of combining the disadvantages of UTF-8 (variable length-encoding) and UTF-32 (not endian-agnostic).
        jrochkind1 6y ago
        Perhaps unicode wouldn't be nearly as successfully adopted as it is, if they had left UCS-2 adopters hanging instead of providing them a "backward compatibility hack" path. The UCS-2 adopters after all had been faithfully trying to implement the standard at that time. Among other things, showing implementers that if they choose to adopt, you aren't going to leave them hanging out to dry when you realize you made a mistake in the standard, will give other people more confidence to adopt. But also, just generally I think a lesson of unicode's success -- as illustrated by UTF-8 in particular -- is, you have to give people a feasible path from where they are to adoption, this is a legitimate part of the design goals of a standard.
- jhasse 6y ago
  Windows finally added support for UTF-8 2 years ago: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/uwp/design/globaliz...
  - ChrisSD 6y ago
    Technically it's had support since (IIRC) Windows 7. What this does is call the translation functions for you instead of having to do it yourself.
    - huhtenberg 6y ago
      Conversion functions - MultiByteToWideChar & co. - were in since Windows 2000 and the UTF8 codepage was supported as early as XP if not in W2K as well.
      - ChrisSD 6y ago
        Ah thanks! It's funny because I can't recall ever using code page 65001 before 7. Maybe there was a reason for that or maybe I simply didn't know it existed until then. Or maybe I thought it simpler to just use UTF-16. I can't remember.
      - gpvos 6y ago
        It existed in W2K and maybe even earlier, but there were bugs in the console regarding codepage 65001, so you couldn't use it as the default. This was not fixed yet in XP, maybe in 7 though.
  - vbezhenar 6y ago
    It is not default configuration and it's marked as "experimental" in UI. I would never enable it for my PC, that's just absurd.
    - iudqnolq 6y ago
      Yeah, the "might" here is doing a lot of work > As Windows operates natively in UTF-16 (WCHAR), you might need to convert UTF-8 data to UTF-16 (or vice versa) to interoperate with Windows APIs.
- coliveira 6y ago
  Java is still using UTF-16, it is the internal format used since its creation. I don't know exactly how much this is a problem or not, but it shows that UTF-16 is still an important thing.
  - rcoveson 6y ago
    I think its a huge problem for Java. Try doing proper string collation (standard library or ICU4J), or regular expression matching, in a context where your strings are all UTF-8 and your output should also be UTF-8. Operations that shouldn't require allocation do, because you have to transcode to UTF-16. Not to mention that in some cases, that transcoding is the most expensive part of the operation. All the core Java APIs are built around String or CharSequence (more the latter in releases post-Java 8). CharSequence is a terrible interface for supporting UTF-8 or any encoding besides latin1 or UTF-16. If Java's interfaces had been designed around Unicode codepoint iteration rather than char random access, then the coupling to UTF-16 wouldn't have been so tight. But as things stand, you aren't doing anything interesting to text in Java without either (1) re-implementing everything from scratch, from integer parsing to regexp, or (2) paying the transcode cost on everything your program consumes and emits.
  - cryptonector 6y ago
    It's a huge problem. UTF-16 is a big big pain. JavaScript (ECMAScript) too has this problem.
julian37 6y ago
UTF-8 is just... so well designed. One little feature I like in particular is that if you're looking for an ASCII-7 character in a UTF-8 stream -- say, a LF or comma -- you don't have to decode the stream first because all bytes in the encoding of non-ASCII-7 characters have the high bit set. Or as Wikipedia puts it: > Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as / (slash) in filenames, \ (backslash) in escape sequences, and % in printf. It's amazing to hear they put it together in one night at a diner! :-D
- losvedir 6y ago
  Yeah, this is great! I came across that recently when working on a parser in Zig, which treats strings as arrays of bytes. I didn't know much about UTF8 other than that it's scary and programmers mess up text processing all the time. I was worried that a multi byte code point could trick my simple char switch which was looking for certain ASCII characters. But then I came across that bit you quoted and was but surprised and relieved! Then, when I needed to minimally handle non-ASCII characters I found Zig's minimal unicode helper library and saw what I was looking for in a small function that takes a leading byte and returns how many bytes there are in the codepoint. I was impressed with the spec again!
- foobarian 6y ago
  > It's amazing to hear they put it together in one night at a diner! :-D I guess you're saying that in good humor. But I'll add this because it makes me appreciate how these things happen: > What happened was this. We had used the original UTF from ISO 10646 to make Plan 9 support 16-bit characters, but we hated it. "We hated it" -- there is just so much going on in those 3 words. They could have been suffering with the previous state for a year for all we know. And even if not, to know you hate something just takes a lot of system building experience to get to. And then when opportunity struck they probably already had a laundry list of grievances they had built up over that time and were ready to pounce.
  - danesparza 6y ago
tkgally 6y ago
> China and Japan could have stayed with their own encodings. Absolutely correct. There was a big debate in Japan in the 1990s about character encodings, with some people arguing strongly against the adoption of Unicode. Their main argument, as I remember it, was that Unicode didn’t capture all of the variations in kanji, especially for personal names. For those of us who were trying to use Japanese online at the time, though, those arguments seemed beside the point. While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1], we were faced with the daily frustration of trying to convert between JIS, S-JIS, EUC, and other encodings and often not being able to exchange Japanese text at all with people who hadn’t installed special software on their computers. It was a great relief when UTF-8 became adopted universally. And now we have emoji, too! [1] https://www.fujitv-view.jp/gallery/post-149246/?imgid=1 https://www.fujitv-view.jp/gallery/post-149246/?imgid=1
- ant6n 6y ago
  > While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1],... Unicode has gotten so big, isn’t this included by now?
  - bmn__ 6y ago
    http://enwp.org/Z-variant http://enwp.org/Z-variant
    - lifthrasiir 6y ago
      Also see the IVD [1]. Indeed both 邉 (U+9089) and 邊 (U+908A) are exceptionally variable characters, the first having 32 variation sequences (the record as of 2020-11-06) and the second having 21 variation sequences. [1] https://unicode.org/ivd/ https://unicode.org/ivd/
- anthk 6y ago
  Does the Japanese goverment use Hiragana on official documments in order to properly spell out names easily?
silvestrov 6y ago
It's the same with html and css: people shit on it all the time, but this just shows they don't have the imagination to see how much worse it could be. Just compare to e.g. Photoshop file format: https://github.com/gco/xee/blob/master/XeePhotoshopLoader.m#L108 https://github.com/gco/xee/blob/master/XeePhotoshopLoader.m#...
- greggman3 6y ago
  The photoshop file format is fine for what is. The format to explode your head is the MS Office .doc format.
swiley 6y ago
ttf is also nearly universally supported. Working with text correctly is just really hard so I think people like to reach for stuff that already does it for them.
jrochkind1 6y ago
Yes, thank you for saying so! Unicode often gets a lot of online hate, which frustrates me, as I agree with you -- Unicode in general it is a remarkably succesful standard, technically as well as with regard to adoption. It's adoption success isn't a coincidence, it's a result of choices made in the design -- with UTF-8 being a big part of that. The choices sometimes involve trade-offs, which lead to the things people complain about (say, the two different codepoint arrangements which can be a é -- there's a reason for that, again related to easing the on-ramp to unicode from legacy technologies, as one of the main goals of UTF-8). There are always trade-offs, nothing is perfect. But Unicode sometimes seems to me to be almost the optimal balance of all the different concerns, I think they could hardly have done better! The "UCS=>UTF-16" mis-step was unfortunate, and we are still dealing with some of the consequences (Java/Windows)... but the fact that we made it through with Unicode adoption only continuing to grow, is a testament to Unicode's good design again. It's not until I ran into some of the "backwaters" of Unicode, realizing they had thought out and specified how to do things like "case-insensitive" normalized collation/comparison for a variety of different specifications in a localized and reasonably performant way... We are so lucky for Unicode.
- selfhoster11 6y ago
  Hate against Unicode frustrates me too. It's like people are privileged and don't know what it's like to have at least 2 code pages for your language. I _still_ have to fix files from those encodings and back into Unicode. Unicode is a blessing even if many don't see it.
chubot 6y ago
Sort of related: I learned from reading about Facebook's lack of moderation that Myanmar is one of the few countries that doesn't use Unicode (and hence UTF-8). It uses something called Zawgyi that apparently has to be heuristically detected! https://en.wikipedia.org/wiki/Myanmar_(Unicode_block)#Historic_and_nonstandard_uses_of_range https://en.wikipedia.org/wiki/Myanmar_(Unicode_block)#Histor... https://www.globalapptesting.com/blog/zawgyi-vs-unicode https://www.globalapptesting.com/blog/zawgyi-vs-unicode
- earthboundkid 6y ago
  Facebook is despicable and indefensible. They knew that they could not moderate Myanmar. They knew or should have known that it was a volatile political situation. The amount of money involved could not have been more than a few million dollars. They should have just turned everything off and said we'll come back when we can. It's disgusting what they did and they should never be forgiven for putting market position ahead of human lives.

roblabla 6y ago

> UTF-16 is still lurking here and there As someone currently stuck in the windows world, this hurts. Every single Windows API is still stuck with using UTF-16/UCS2 as the string encoding. Also fun fact, on the Nintendo Switch, various subsystems use different kind of encoding. The filsystem submodule uses Shift-JIS, most of the other modules use UTF-8, but some others yet use UTF-16 (like the virtual keyboard, IIRC). A brilliant mess.

ChrisSD 6y ago
Personally I don't find UTF-16 to be too bad. It's a simple encoding and very easy to convert to/from UTF-8. So your program can be written in UTF-8 and your WinAPI wrappers can convert as/when needed.
- jeltz 6y ago
  The bad thing with UTF-16 is that so much software assumes that one code point always is 16 bits.
  - cygx 6y ago
    Most of the hard stuff is there no matter the encoding (normalization, user-perceived characters spanning multiple code units, paths vs strings, ...).
  - jrochkind1 6y ago
    Which is not UTF-16 at all, UTF-16 standard clearly says this is not so. So why do they do that? It's actually a leftover of the earlier UCS-2 standard, before it was realized we'd need more codepoints than that, and that it was a mistake to limit to 16-bit space for codepoints in any encoding. Software written for UCS-2 can mostly work compatibly with UTF-16, but there are some problems, encoding the 'higher' codepoints is only one of several. Another is how right-to-left scripts are handled. http://www.differencebetween.net/technology/software-technology/difference-between-ucs-2-and-utf-16/ http://www.differencebetween.net/technology/software-technol... https://unicode.org/faq/utf_bom.html#utf16-11 https://unicode.org/faq/utf_bom.html#utf16-11
    - flohofwoe 6y ago
      Wasn't UTF-16 explicitly created as a "backward compatibility hack" for UCS-2 when it became clear that 16 bits per code point isn't enough? They should have ditched 16-bit encodings back then instead of combining the disadvantages of UTF-8 (variable length-encoding) and UTF-32 (not endian-agnostic).
      - jrochkind1 6y ago
        Perhaps unicode wouldn't be nearly as successfully adopted as it is, if they had left UCS-2 adopters hanging instead of providing them a "backward compatibility hack" path. The UCS-2 adopters after all had been faithfully trying to implement the standard at that time. Among other things, showing implementers that if they choose to adopt, you aren't going to leave them hanging out to dry when you realize you made a mistake in the standard, will give other people more confidence to adopt. But also, just generally I think a lesson of unicode's success -- as illustrated by UTF-8 in particular -- is, you have to give people a feasible path from where they are to adoption, this is a legitimate part of the design goals of a standard.
jhasse 6y ago
Windows finally added support for UTF-8 2 years ago: https://docs.microsoft.com/en-us/windows/uwp/design/globalizing/use-utf8-code-page https://docs.microsoft.com/en-us/windows/uwp/design/globaliz...
- ChrisSD 6y ago
  Technically it's had support since (IIRC) Windows 7. What this does is call the translation functions for you instead of having to do it yourself.
  - huhtenberg 6y ago
    Conversion functions - MultiByteToWideChar & co. - were in since Windows 2000 and the UTF8 codepage was supported as early as XP if not in W2K as well.
    - ChrisSD 6y ago
      Ah thanks! It's funny because I can't recall ever using code page 65001 before 7. Maybe there was a reason for that or maybe I simply didn't know it existed until then. Or maybe I thought it simpler to just use UTF-16. I can't remember.
    - gpvos 6y ago
      It existed in W2K and maybe even earlier, but there were bugs in the console regarding codepage 65001, so you couldn't use it as the default. This was not fixed yet in XP, maybe in 7 though.
- vbezhenar 6y ago
  It is not default configuration and it's marked as "experimental" in UI. I would never enable it for my PC, that's just absurd.
  - iudqnolq 6y ago
    Yeah, the "might" here is doing a lot of work > As Windows operates natively in UTF-16 (WCHAR), you might need to convert UTF-8 data to UTF-16 (or vice versa) to interoperate with Windows APIs.
coliveira 6y ago
Java is still using UTF-16, it is the internal format used since its creation. I don't know exactly how much this is a problem or not, but it shows that UTF-16 is still an important thing.
- rcoveson 6y ago
  I think its a huge problem for Java. Try doing proper string collation (standard library or ICU4J), or regular expression matching, in a context where your strings are all UTF-8 and your output should also be UTF-8. Operations that shouldn't require allocation do, because you have to transcode to UTF-16. Not to mention that in some cases, that transcoding is the most expensive part of the operation. All the core Java APIs are built around String or CharSequence (more the latter in releases post-Java 8). CharSequence is a terrible interface for supporting UTF-8 or any encoding besides latin1 or UTF-16. If Java's interfaces had been designed around Unicode codepoint iteration rather than char random access, then the coupling to UTF-16 wouldn't have been so tight. But as things stand, you aren't doing anything interesting to text in Java without either (1) re-implementing everything from scratch, from integer parsing to regexp, or (2) paying the transcode cost on everything your program consumes and emits.
- cryptonector 6y ago
  It's a huge problem. UTF-16 is a big big pain. JavaScript (ECMAScript) too has this problem.

julian37 6y ago

UTF-8 is just... so well designed. One little feature I like in particular is that if you're looking for an ASCII-7 character in a UTF-8 stream -- say, a LF or comma -- you don't have to decode the stream first because all bytes in the encoding of non-ASCII-7 characters have the high bit set. Or as Wikipedia puts it: > Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as / (slash) in filenames, \ (backslash) in escape sequences, and % in printf. It's amazing to hear they put it together in one night at a diner! :-D

losvedir 6y ago
Yeah, this is great! I came across that recently when working on a parser in Zig, which treats strings as arrays of bytes. I didn't know much about UTF8 other than that it's scary and programmers mess up text processing all the time. I was worried that a multi byte code point could trick my simple char switch which was looking for certain ASCII characters. But then I came across that bit you quoted and was but surprised and relieved! Then, when I needed to minimally handle non-ASCII characters I found Zig's minimal unicode helper library and saw what I was looking for in a small function that takes a leading byte and returns how many bytes there are in the codepoint. I was impressed with the spec again!
foobarian 6y ago
> It's amazing to hear they put it together in one night at a diner! :-D I guess you're saying that in good humor. But I'll add this because it makes me appreciate how these things happen: > What happened was this. We had used the original UTF from ISO 10646 to make Plan 9 support 16-bit characters, but we hated it. "We hated it" -- there is just so much going on in those 3 words. They could have been suffering with the previous state for a year for all we know. And even if not, to know you hate something just takes a lot of system building experience to get to. And then when opportunity struck they probably already had a laundry list of grievances they had built up over that time and were ready to pounce.
- danesparza 6y ago

tkgally 6y ago

> China and Japan could have stayed with their own encodings. Absolutely correct. There was a big debate in Japan in the 1990s about character encodings, with some people arguing strongly against the adoption of Unicode. Their main argument, as I remember it, was that Unicode didn’t capture all of the variations in kanji, especially for personal names. For those of us who were trying to use Japanese online at the time, though, those arguments seemed beside the point. While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1], we were faced with the daily frustration of trying to convert between JIS, S-JIS, EUC, and other encodings and often not being able to exchange Japanese text at all with people who hadn’t installed special software on their computers. It was a great relief when UTF-8 became adopted universally. And now we have emoji, too! [1] https://www.fujitv-view.jp/gallery/post-149246/?imgid=1 https://www.fujitv-view.jp/gallery/post-149246/?imgid=1

ant6n 6y ago
> While it would have been nice, in an ideal world, to be able to encode and display all 51 (at least) variations in the second kanji for the surname Watanabe [1],... Unicode has gotten so big, isn’t this included by now?
- bmn__ 6y ago
  http://enwp.org/Z-variant http://enwp.org/Z-variant
  - lifthrasiir 6y ago
    Also see the IVD [1]. Indeed both 邉 (U+9089) and 邊 (U+908A) are exceptionally variable characters, the first having 32 variation sequences (the record as of 2020-11-06) and the second having 21 variation sequences. [1] https://unicode.org/ivd/ https://unicode.org/ivd/
anthk 6y ago
Does the Japanese goverment use Hiragana on official documments in order to properly spell out names easily?

silvestrov 6y ago

It's the same with html and css: people shit on it all the time, but this just shows they don't have the imagination to see how much worse it could be. Just compare to e.g. Photoshop file format: https://github.com/gco/xee/blob/master/XeePhotoshopLoader.m#L108 https://github.com/gco/xee/blob/master/XeePhotoshopLoader.m#...

greggman3 6y ago
The photoshop file format is fine for what is. The format to explode your head is the MS Office .doc format.

swiley 6y ago

ttf is also nearly universally supported. Working with text correctly is just really hard so I think people like to reach for stuff that already does it for them.

jrochkind1 6y ago

Yes, thank you for saying so! Unicode often gets a lot of online hate, which frustrates me, as I agree with you -- Unicode in general it is a remarkably succesful standard, technically as well as with regard to adoption. It's adoption success isn't a coincidence, it's a result of choices made in the design -- with UTF-8 being a big part of that. The choices sometimes involve trade-offs, which lead to the things people complain about (say, the two different codepoint arrangements which can be a é -- there's a reason for that, again related to easing the on-ramp to unicode from legacy technologies, as one of the main goals of UTF-8). There are always trade-offs, nothing is perfect. But Unicode sometimes seems to me to be almost the optimal balance of all the different concerns, I think they could hardly have done better! The "UCS=>UTF-16" mis-step was unfortunate, and we are still dealing with some of the consequences (Java/Windows)... but the fact that we made it through with Unicode adoption only continuing to grow, is a testament to Unicode's good design again. It's not until I ran into some of the "backwaters" of Unicode, realizing they had thought out and specified how to do things like "case-insensitive" normalized collation/comparison for a variety of different specifications in a localized and reasonably performant way... We are so lucky for Unicode.

selfhoster11 6y ago
Hate against Unicode frustrates me too. It's like people are privileged and don't know what it's like to have at least 2 code pages for your language. I _still_ have to fix files from those encodings and back into Unicode. Unicode is a blessing even if many don't see it.

chubot 6y ago

Sort of related: I learned from reading about Facebook's lack of moderation that Myanmar is one of the few countries that doesn't use Unicode (and hence UTF-8). It uses something called Zawgyi that apparently has to be heuristically detected! https://en.wikipedia.org/wiki/Myanmar_(Unicode_block)#Historic_and_nonstandard_uses_of_range https://en.wikipedia.org/wiki/Myanmar_(Unicode_block)#Histor... https://www.globalapptesting.com/blog/zawgyi-vs-unicode https://www.globalapptesting.com/blog/zawgyi-vs-unicode

earthboundkid 6y ago
Facebook is despicable and indefensible. They knew that they could not moderate Myanmar. They knew or should have known that it was a volatile political situation. The amount of money involved could not have been more than a few million dollars. They should have just turned everything off and said we'll come back when we can. It's disgusting what they did and they should never be forgiven for putting market position ahead of human lives.