5 ms·

What's not to understand about UTF8? It's a way of coding up points in the Unicode space. You decode the bits and it yields a number that corresponds to a glyph

What's not to understand about UTF8? It's a way of coding up points in the Unicode space. You decode the bits and it yields a number that corresponds to a glyph, possibly with some decorations. The only thing special about UTF8 is that it happens to write ASCII as ASCII, which is nice as long as you realize that there is much outside that. Either that, or I'm completely off base and part of the vast horde who don't get it.

henriquez 3y ago
I read your comment in the “Simpson comic book nerd” tone of voice.
Dylan16807 3y ago
Variable width encodings are inherently harder to understand. And while UTF-8 tries very hard to be simple whenever possible, it does have some nonobvious constraints that make it significantly more complex than the actual simplest variable-width encoding (a continuation bit and then 7 data bits).
- morelisp 3y ago
  Even ignoring all the other advantages (mostly synchronization-related, which do objectively make implementing algorithms on UTF-8-encoded data simpler), "the number of set prefix bits is the number of bytes" doesn't seem meaningfully more complex than a single continuation bit.
  - Dylan16807 3y ago
    > "the number of set prefix bits is the number of bytes" Except when it's 1, because that's an invalid start, and except when it's 0, because that means the character is a single byte. And it also means you're dealing with three classes of byte now. Plus UTF-8 has more invalid encodings to deal with than a super-simple format.
    - morelisp 3y ago
      > Plus UTF-8 has more invalid encodings to deal with than a super-simple format. If your format supports non-canonical encodings you're in for a bad time no matter what, so a whole lot of that simplicity is fake. > And it also means you're dealing with three classes of byte now. If you're working a byte at a time you're doing it wrong, unless you're re-syncing an invalid stream in which case it's as simple as a continuation bit (specifically, it's two continuation bits).
      - Dylan16807 3y ago
        The simple encoding already allows smaller characters to have the same bytes as subsets of larger characters. Non-canonical is not a big deal on top of that. Also there are other banned bytes you don't need to deal with. > If you're working a byte at a time you're doing it wrong, unless you're re-syncing an invalid stream It's very relevant to explaining the encoding and it matters if you're worried that invalid bytes might exist. You can't just ignore the extra complexity. Also if you're not working a byte at a time, that kind of implies you parsed the characters? In which case non-canonical encodings are a non-problem.
        morelisp 3y ago
        > Non-canonical is not a big deal on top of that. Unless you want to actually do anything with the string beyond decode a codepoint.
        Dylan16807 3y ago
        If you're going beyond decoding, then you're beyond the stage where canonical and non-canonical versions exist any more. Non-canonical encodings make it difficult to do things without decoding, but you have bigger problems to deal with in that situation, and the non-canonical encodings don't make it much worse. Don't get into that situation! Specifically, even with only canonical encodings, one and two byte characters can appear inside the encoding of two and three byte characters. You can't do anything byte-wise at all, unlike UTF-8. But you already said "If you're working a byte at a time you're doing it wrong" so I hope that's not too big of an issue?
- asveikau 3y ago
  Unicode is variable length even if you use 32 bits, because glyphs sometimes require multiple codepoints. People sometimes write as if using more bits will remove complexity from Unicode but it doesn't really, you still need to handle multiple units at once sometimes.
  - Dylan16807 3y ago
    Sure but that's at another level entirely. Dealing with two types of variable width is double difficult.
    - asveikau 3y ago
      I disagree, if you are correctly handling Unicode you already are going through some "decode" function which parses a 32 bit quantity from utf-8 or utf-16. From there you need to handle multiple codepoints together sometimes, like for non-composed diacritics, han unification, and some emojis. This is complex regardless of whether or not you use utf-8 or utf-16, in fact I'd say it's more difficult to handle than those.
      - Dylan16807 3y ago
        We're not talking about fully correctly handling Unicode, we're talking about teaching someone the basics of an encoding. Context is critical here.
        asveikau 3y ago
        If you want to teach someone how an encoding works, why would you not tell them that a single symbol can take multiple codepoints? It seems like you're advocating people learning incorrect information and forming their impressions of it on falsehood. Which is probably why people think utf-32 frees you from variable-length encoding.
        Dylan16807 3y ago
        You should tell them, yes. And talk about it more at some point. But you don't have to go into much detail when today's lesson is specifically teaching UTF-8 or UTF-32. I don't know about you, but I think I could teach the latter about ten times faster. As part of a comprehensive dive into Unicode it's a minor part, but for teaching an encoding it's a significant difference.
nayuki 3y ago
More properties of UTF-8: It is self-synchronizing. It has the same lexicographical sort order as UTF-32. It allows substring matches without false positives. It is compatible with null-terminated strings.

henriquez 3y ago

I read your comment in the “Simpson comic book nerd” tone of voice.

Dylan16807 3y ago

Variable width encodings are inherently harder to understand. And while UTF-8 tries very hard to be simple whenever possible, it does have some nonobvious constraints that make it significantly more complex than the actual simplest variable-width encoding (a continuation bit and then 7 data bits).

morelisp 3y ago
Even ignoring all the other advantages (mostly synchronization-related, which do objectively make implementing algorithms on UTF-8-encoded data simpler), "the number of set prefix bits is the number of bytes" doesn't seem meaningfully more complex than a single continuation bit.
- Dylan16807 3y ago
  > "the number of set prefix bits is the number of bytes" Except when it's 1, because that's an invalid start, and except when it's 0, because that means the character is a single byte. And it also means you're dealing with three classes of byte now. Plus UTF-8 has more invalid encodings to deal with than a super-simple format.
  - morelisp 3y ago
    > Plus UTF-8 has more invalid encodings to deal with than a super-simple format. If your format supports non-canonical encodings you're in for a bad time no matter what, so a whole lot of that simplicity is fake. > And it also means you're dealing with three classes of byte now. If you're working a byte at a time you're doing it wrong, unless you're re-syncing an invalid stream in which case it's as simple as a continuation bit (specifically, it's two continuation bits).
    - Dylan16807 3y ago
      The simple encoding already allows smaller characters to have the same bytes as subsets of larger characters. Non-canonical is not a big deal on top of that. Also there are other banned bytes you don't need to deal with. > If you're working a byte at a time you're doing it wrong, unless you're re-syncing an invalid stream It's very relevant to explaining the encoding and it matters if you're worried that invalid bytes might exist. You can't just ignore the extra complexity. Also if you're not working a byte at a time, that kind of implies you parsed the characters? In which case non-canonical encodings are a non-problem.
      - morelisp 3y ago
        > Non-canonical is not a big deal on top of that. Unless you want to actually do anything with the string beyond decode a codepoint.
        Dylan16807 3y ago
        If you're going beyond decoding, then you're beyond the stage where canonical and non-canonical versions exist any more. Non-canonical encodings make it difficult to do things without decoding, but you have bigger problems to deal with in that situation, and the non-canonical encodings don't make it much worse. Don't get into that situation! Specifically, even with only canonical encodings, one and two byte characters can appear inside the encoding of two and three byte characters. You can't do anything byte-wise at all, unlike UTF-8. But you already said "If you're working a byte at a time you're doing it wrong" so I hope that's not too big of an issue?
asveikau 3y ago
Unicode is variable length even if you use 32 bits, because glyphs sometimes require multiple codepoints. People sometimes write as if using more bits will remove complexity from Unicode but it doesn't really, you still need to handle multiple units at once sometimes.
- Dylan16807 3y ago
  Sure but that's at another level entirely. Dealing with two types of variable width is double difficult.
  - asveikau 3y ago
    I disagree, if you are correctly handling Unicode you already are going through some "decode" function which parses a 32 bit quantity from utf-8 or utf-16. From there you need to handle multiple codepoints together sometimes, like for non-composed diacritics, han unification, and some emojis. This is complex regardless of whether or not you use utf-8 or utf-16, in fact I'd say it's more difficult to handle than those.
    - Dylan16807 3y ago
      We're not talking about fully correctly handling Unicode, we're talking about teaching someone the basics of an encoding. Context is critical here.
      - asveikau 3y ago
        If you want to teach someone how an encoding works, why would you not tell them that a single symbol can take multiple codepoints? It seems like you're advocating people learning incorrect information and forming their impressions of it on falsehood. Which is probably why people think utf-32 frees you from variable-length encoding.
        Dylan16807 3y ago
        You should tell them, yes. And talk about it more at some point. But you don't have to go into much detail when today's lesson is specifically teaching UTF-8 or UTF-32. I don't know about you, but I think I could teach the latter about ten times faster. As part of a comprehensive dive into Unicode it's a minor part, but for teaching an encoding it's a significant difference.

nayuki 3y ago

More properties of UTF-8: It is self-synchronizing. It has the same lexicographical sort order as UTF-32. It allows substring matches without false positives. It is compatible with null-terminated strings.