5 ms·

Utf8 is one of the most momentous and under appreciated / relatively unknown achievements in software. A sketch on a diner placemat has lead to every person in

by bhawks 5y ago

Utf8 is one of the most momentous and under appreciated / relatively unknown achievements in software. A sketch on a diner placemat has lead to every person in the world being able to communicate written language digitally using a common software stack. Thanks to Ken Thompson and Rob Pike we have avoided the deeply siloed and incompatible world that code pages, wide chars and other insufficient encoding schemes were guiding us towards.

ahelwer 5y ago
It really is wonderful. I was forced to wrap my head around it in the past year while writing a tree-sitter grammar for a language that supports Unicode. Calculating column position gets a whole lot trickier when the preceding codepoints are of variable byte-width! It's one of those rabbit holes where you can see people whose entire career is wrapped up in incredibly tiny details like what number maps to what symbol - and it can get real political!
deleted 5y ago
[deleted]
inglor_cz 5y ago
As a young Czech programming acolyte in the late 1990s, I had to cope with several competing 8-bit encodings. It was a pure nightmare. Long live UTF-8. Finally I can write any Central European name without mutilating it.
cryptonector 5y ago
And stayed ASCII-compatible. And did not have to go to wide chars. And it does not suck. And it resynchronizes. And...
- wolverine876 5y ago
  > And did not have to go to wide chars. What do you mean by that? Unicode does have double-wide characters and, I discovered recently, some other characters called 'wide' that at least are wider than one column but smaller than two in at least some monospaced fonts. Try: * Small Hyphen-minus (U+FE63) "﹣": Seems to be >1 and <2 columns in at least some monospaced fonts. * Fullwidth Hyphen-minus (U+FF0D) "－": ditto
  - dahfizz 5y ago
    Some Unicode encodings require a char datatype that is 2+ bytes wide, hence a wide char.
    - wolverine876 5y ago
      Right, so what does the GGP mean by "did not have to go to wide chars"?
      - dahfizz 5y ago
        Utf-8 can be implemented with plain old 8-bit chars. It does not need the wide chars like UCS-2, for example, where every character is 16 bytes. Therefore, by going to utf8 instead of ucs2, we did not have to go to wide chars.
      - cryptonector 5y ago
        "wide chars" == wchar_t, sizeof(char) == 1, sizeof(wchar_t) > sizeof(char).
      - masklinn 5y ago
        The code unit of UTF8 is the byte, so you only deal with bytes and byte sequences. Thus no issues of byte-order and such, which you have to deal with in UTF-16 and UTF-32 (as their code units are respectively 2 and 4 bytes).
GekkePrutser 5y ago
It's great as a global character set and really enabled the world to move ahead at just the right time when the web started to connect us all together. But the whole emoji modifier (e.g. guy + heart + lips + girl = one kissing couple character) thing is a disaster. Too many rules made up on the fly that make building an accurate parser a nightmare. It should have either specified this strictly and consistently as part of the standard, or just left it out for a future standard to implement, and just just used separate codepoints for the combinations that were really necessary. This complexity is also something that has led to multiple vulnerabilities especially on mobiles. See here all the combos: https://unicode.org/emoji/charts/full-emoji-modifiers.html https://unicode.org/emoji/charts/full-emoji-modifiers.html
- zerocrates 5y ago
  Whatever you may think of emoji and all the attendant baggage that comes with them doesn't really have any bearing on how great UTF-8 is.
  - GekkePrutser 5y ago
    I have nothing against emoji. Just the combination mechanism. And yeah this problem only came later after UTF-8 was already invented. It's a smart solution also because at the time a lot of detractors of Unicode were opposed to the extra data requirements of the long form representations.
    - saagarjha 5y ago
      The combination mechanism is a core part of Unicode, not just a thing added to make emoji work.
- WorldMaker 5y ago
  Emoji are not any more complex than some natural languages that people need to encode in Unicode. It's a weird, more modern "language" than most, but it's a still a human language with human quirks. It's a lovely privilege that English speakers generally don't have any language encoding problems to deal with beyond Emoji, but it's a brilliant bit of Unicode today that Emoji is becoming a required by users common denominator that stress tests incorrect assumptions English speakers want to make about language encoding.
yongjik 5y ago
UTF-8 is a great system, but all those dreadful code pages existed because they were under different technical constraints. Windows machines in 1990s had several megabytes of main memory, and people could barely get it to support one East Asian language at a time, never mind multiple of them. No sane person would propose using three bytes per a Korean character when two would do - that would mean your word processor will die after adding 50 pages of document, while your competitor can do 75. And even if you did have UTF-8, you wouldn't see those Thai characters anyway, because who would even have these fonts when your OS must fit in a handful of stacked floppies. It took years before UTF-8 made technical sense for most users.
- chrismorgan 5y ago
  Why would you keep 50 pages of document in main memory at once? It’s not like 75 is some magic limit that’s enough and 50 isn’t. No, if you stand any chance of getting anywhere near such a limit, you would certainly design your data structures so you don’t need all the content in memory at once, and so the difference is not so ferociously significant.
  - deleted 5y ago
    [deleted]
  - dahfizz 5y ago
    It's not like disk space was free and limitless, as was pointed out. Bytes in general used to be way way way more expensive and precious.
    - chrismorgan 5y ago
      It wasn’t free and limitless, but it wasn’t scarce either—you probably had 100–1000× more disk space than RAM, which is close enough to unlimited for most text purposes. (https://en.wikipedia.org/wiki/History_of_hard_disk_drives https://en.wikipedia.org/wiki/History_of_hard_disk_drives suggests 1GB was typical in the mid-1990s.) Consider also that at this very time we’re talking of (early 1990s) the industry was shifting away from largely 8-bit code pages to 16-bit UCS-2, which is an even more extreme cost when compared to UTF-8, doubling space requirements for most people, rather than merely the 50% increase yongjik speaks of for certain languages. Yet this change was being done (more’s the pity). Concerning the scarcity of bytes, yongjik’s point would certainly be valid if it referred to the 1970s, was probably valid of the 1980s, but is not valid of the 1990s. (But the point about keeping the full document in RAM is an unrealistic strawman.)
StefanKarpinski 5y ago
This post is a really good illustration of UTF-8. Very clear! The key brilliance of the design is not only the embedding of ASCII in UTF-8, but the fact that nothing in ASCII can appear anywhere else UTF-8, and more generally that no UTF-8 character can appear as a substring of another character’s encoding. That means that all the byte-oriented libc string functions just work. I wrote this up recently in a StackOverflow answer with some examples: https://stackoverflow.com/a/69756619/659248 https://stackoverflow.com/a/69756619/659248
- wolverine876 5y ago
  > nothing in ASCII can appear anywhere else UTF-8, and more generally that no UTF-8 character can appear as a substring of another character’s encoding How is that defined and enforced? Very narrowly, it seems to me: * ASCII Hyphen-minus (U+002D) has similar functions and appearance to Small Hyphen-minus (U+FE63), Fullwidth Hyphen-minus (U+FF0D), Hyphen (U+2010), Minus Sign (U+2212), Heavy Minus Sign (U+2796), En dash (U+2013), Em Dash (U+2014), Small Em Dash (U+FE58), Horizontal Bar (U+2015), Figure dash (U+2012). (I'm probably missing a few!) * There are separate delta symbols for Greek and for mathematics (sorry, no more time for looking up code points). * Very many other characters have appearances so similar that nobody could tell them apart. So characters have apparently, to users and almost anyone not looking at the actual codes, identical functions and appearances.
  - dahfizz 5y ago
    OP (and the article) is talking about the encoding of UTF-8, not Unicode in general. ASCII is itself valid utf-8, because ASCII is a subset of utf-8. But a multi byte encoded codepoint in UTF-8 cannot be confused with ASCII, because the highest bit is set in all the octets.

by bhawks 5y ago

ahelwer 5y ago

It really is wonderful. I was forced to wrap my head around it in the past year while writing a tree-sitter grammar for a language that supports Unicode. Calculating column position gets a whole lot trickier when the preceding codepoints are of variable byte-width! It's one of those rabbit holes where you can see people whose entire career is wrapped up in incredibly tiny details like what number maps to what symbol - and it can get real political!

deleted 5y ago

[deleted]

inglor_cz 5y ago

As a young Czech programming acolyte in the late 1990s, I had to cope with several competing 8-bit encodings. It was a pure nightmare. Long live UTF-8. Finally I can write any Central European name without mutilating it.

cryptonector 5y ago

And stayed ASCII-compatible. And did not have to go to wide chars. And it does not suck. And it resynchronizes. And...

wolverine876 5y ago
> And did not have to go to wide chars. What do you mean by that? Unicode does have double-wide characters and, I discovered recently, some other characters called 'wide' that at least are wider than one column but smaller than two in at least some monospaced fonts. Try: * Small Hyphen-minus (U+FE63) "﹣": Seems to be >1 and <2 columns in at least some monospaced fonts. * Fullwidth Hyphen-minus (U+FF0D) "－": ditto
- dahfizz 5y ago
  Some Unicode encodings require a char datatype that is 2+ bytes wide, hence a wide char.
  - wolverine876 5y ago
    Right, so what does the GGP mean by "did not have to go to wide chars"?
    - dahfizz 5y ago
      Utf-8 can be implemented with plain old 8-bit chars. It does not need the wide chars like UCS-2, for example, where every character is 16 bytes. Therefore, by going to utf8 instead of ucs2, we did not have to go to wide chars.
    - cryptonector 5y ago
      "wide chars" == wchar_t, sizeof(char) == 1, sizeof(wchar_t) > sizeof(char).
    - masklinn 5y ago
      The code unit of UTF8 is the byte, so you only deal with bytes and byte sequences. Thus no issues of byte-order and such, which you have to deal with in UTF-16 and UTF-32 (as their code units are respectively 2 and 4 bytes).

GekkePrutser 5y ago

It's great as a global character set and really enabled the world to move ahead at just the right time when the web started to connect us all together. But the whole emoji modifier (e.g. guy + heart + lips + girl = one kissing couple character) thing is a disaster. Too many rules made up on the fly that make building an accurate parser a nightmare. It should have either specified this strictly and consistently as part of the standard, or just left it out for a future standard to implement, and just just used separate codepoints for the combinations that were really necessary. This complexity is also something that has led to multiple vulnerabilities especially on mobiles. See here all the combos: https://unicode.org/emoji/charts/full-emoji-modifiers.html https://unicode.org/emoji/charts/full-emoji-modifiers.html

zerocrates 5y ago
Whatever you may think of emoji and all the attendant baggage that comes with them doesn't really have any bearing on how great UTF-8 is.
- GekkePrutser 5y ago
  I have nothing against emoji. Just the combination mechanism. And yeah this problem only came later after UTF-8 was already invented. It's a smart solution also because at the time a lot of detractors of Unicode were opposed to the extra data requirements of the long form representations.
  - saagarjha 5y ago
    The combination mechanism is a core part of Unicode, not just a thing added to make emoji work.
WorldMaker 5y ago
Emoji are not any more complex than some natural languages that people need to encode in Unicode. It's a weird, more modern "language" than most, but it's a still a human language with human quirks. It's a lovely privilege that English speakers generally don't have any language encoding problems to deal with beyond Emoji, but it's a brilliant bit of Unicode today that Emoji is becoming a required by users common denominator that stress tests incorrect assumptions English speakers want to make about language encoding.

yongjik 5y ago

UTF-8 is a great system, but all those dreadful code pages existed because they were under different technical constraints. Windows machines in 1990s had several megabytes of main memory, and people could barely get it to support one East Asian language at a time, never mind multiple of them. No sane person would propose using three bytes per a Korean character when two would do - that would mean your word processor will die after adding 50 pages of document, while your competitor can do 75. And even if you did have UTF-8, you wouldn't see those Thai characters anyway, because who would even have these fonts when your OS must fit in a handful of stacked floppies. It took years before UTF-8 made technical sense for most users.

chrismorgan 5y ago
Why would you keep 50 pages of document in main memory at once? It’s not like 75 is some magic limit that’s enough and 50 isn’t. No, if you stand any chance of getting anywhere near such a limit, you would certainly design your data structures so you don’t need all the content in memory at once, and so the difference is not so ferociously significant.
- deleted 5y ago
  [deleted]
- dahfizz 5y ago
  It's not like disk space was free and limitless, as was pointed out. Bytes in general used to be way way way more expensive and precious.
  - chrismorgan 5y ago
    It wasn’t free and limitless, but it wasn’t scarce either—you probably had 100–1000× more disk space than RAM, which is close enough to unlimited for most text purposes. (https://en.wikipedia.org/wiki/History_of_hard_disk_drives https://en.wikipedia.org/wiki/History_of_hard_disk_drives suggests 1GB was typical in the mid-1990s.) Consider also that at this very time we’re talking of (early 1990s) the industry was shifting away from largely 8-bit code pages to 16-bit UCS-2, which is an even more extreme cost when compared to UTF-8, doubling space requirements for most people, rather than merely the 50% increase yongjik speaks of for certain languages. Yet this change was being done (more’s the pity). Concerning the scarcity of bytes, yongjik’s point would certainly be valid if it referred to the 1970s, was probably valid of the 1980s, but is not valid of the 1990s. (But the point about keeping the full document in RAM is an unrealistic strawman.)

StefanKarpinski 5y ago

This post is a really good illustration of UTF-8. Very clear! The key brilliance of the design is not only the embedding of ASCII in UTF-8, but the fact that nothing in ASCII can appear anywhere else UTF-8, and more generally that no UTF-8 character can appear as a substring of another character’s encoding. That means that all the byte-oriented libc string functions just work. I wrote this up recently in a StackOverflow answer with some examples: https://stackoverflow.com/a/69756619/659248 https://stackoverflow.com/a/69756619/659248

wolverine876 5y ago
> nothing in ASCII can appear anywhere else UTF-8, and more generally that no UTF-8 character can appear as a substring of another character’s encoding How is that defined and enforced? Very narrowly, it seems to me: * ASCII Hyphen-minus (U+002D) has similar functions and appearance to Small Hyphen-minus (U+FE63), Fullwidth Hyphen-minus (U+FF0D), Hyphen (U+2010), Minus Sign (U+2212), Heavy Minus Sign (U+2796), En dash (U+2013), Em Dash (U+2014), Small Em Dash (U+FE58), Horizontal Bar (U+2015), Figure dash (U+2012). (I'm probably missing a few!) * There are separate delta symbols for Greek and for mathematics (sorry, no more time for looking up code points). * Very many other characters have appearances so similar that nobody could tell them apart. So characters have apparently, to users and almost anyone not looking at the actual codes, identical functions and appearances.
- dahfizz 5y ago
  OP (and the article) is talking about the encoding of UTF-8, not Unicode in general. ASCII is itself valid utf-8, because ASCII is a subset of utf-8. But a multi byte encoded codepoint in UTF-8 cannot be confused with ASCII, because the highest bit is set in all the octets.