4 ms·

This is a great video! I’ve been writing about UTF-8 on my blog and I noticed that many programmers don’t understand it, after 10 or 20 years The main impedim

by chubot 3y ago

This is a great video! I’ve been writing about UTF-8 on my blog and I noticed that many programmers don’t understand it, after 10 or 20 years The main impediment seems to be that languages like Java, JavaScript, and Python treat UTF-8 as just another encoding, but really it’s the most fundamental encoding. The language abstractions get in the way and distort the way people think about text Newer languages like Go and Rust are more sensible, they don’t have global mutable encoding variables

omsimun 3y ago
It's misleading to describe UTF-8 as "the most fundamental encoding", because of the existence of UTF-32 (essentially just a trivial encoding of "raw" Unicode code points) and UTF-16 (which has certain limitations that later became part of the specification of what code points are valid in any encoding) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surrogates) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surro... UTF-8 is the most ubiquitous encoding on the web, but that doesn't make it more fundamental than any other.
- lifthrasiir 3y ago
  The word "fundamental" here probably has the same meaning as "elementary" as in elementary mathematics, i.e. something you should absolutely understand first, not something that makes up everything else.
jfengel 3y ago
What's not to understand about UTF8? It's a way of coding up points in the Unicode space. You decode the bits and it yields a number that corresponds to a glyph, possibly with some decorations. The only thing special about UTF8 is that it happens to write ASCII as ASCII, which is nice as long as you realize that there is much outside that. Either that, or I'm completely off base and part of the vast horde who don't get it.
- henriquez 3y ago
  I read your comment in the “Simpson comic book nerd” tone of voice.
- Dylan16807 3y ago
  Variable width encodings are inherently harder to understand. And while UTF-8 tries very hard to be simple whenever possible, it does have some nonobvious constraints that make it significantly more complex than the actual simplest variable-width encoding (a continuation bit and then 7 data bits).
  - morelisp 3y ago
    Even ignoring all the other advantages (mostly synchronization-related, which do objectively make implementing algorithms on UTF-8-encoded data simpler), "the number of set prefix bits is the number of bytes" doesn't seem meaningfully more complex than a single continuation bit.
    - Dylan16807 3y ago
      > "the number of set prefix bits is the number of bytes" Except when it's 1, because that's an invalid start, and except when it's 0, because that means the character is a single byte. And it also means you're dealing with three classes of byte now. Plus UTF-8 has more invalid encodings to deal with than a super-simple format.
      - morelisp 3y ago
        > Plus UTF-8 has more invalid encodings to deal with than a super-simple format. If your format supports non-canonical encodings you're in for a bad time no matter what, so a whole lot of that simplicity is fake. > And it also means you're dealing with three classes of byte now. If you're working a byte at a time you're doing it wrong, unless you're re-syncing an invalid stream in which case it's as simple as a continuation bit (specifically, it's two continuation bits).

by chubot 3y ago

omsimun 3y ago

It's misleading to describe UTF-8 as "the most fundamental encoding", because of the existence of UTF-32 (essentially just a trivial encoding of "raw" Unicode code points) and UTF-16 (which has certain limitations that later became part of the specification of what code points are valid in any encoding) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surrogates) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surro... UTF-8 is the most ubiquitous encoding on the web, but that doesn't make it more fundamental than any other.

lifthrasiir 3y ago
The word "fundamental" here probably has the same meaning as "elementary" as in elementary mathematics, i.e. something you should absolutely understand first, not something that makes up everything else.

jfengel 3y ago

What's not to understand about UTF8? It's a way of coding up points in the Unicode space. You decode the bits and it yields a number that corresponds to a glyph, possibly with some decorations. The only thing special about UTF8 is that it happens to write ASCII as ASCII, which is nice as long as you realize that there is much outside that. Either that, or I'm completely off base and part of the vast horde who don't get it.

henriquez 3y ago
I read your comment in the “Simpson comic book nerd” tone of voice.
Dylan16807 3y ago
Variable width encodings are inherently harder to understand. And while UTF-8 tries very hard to be simple whenever possible, it does have some nonobvious constraints that make it significantly more complex than the actual simplest variable-width encoding (a continuation bit and then 7 data bits).
- morelisp 3y ago
  Even ignoring all the other advantages (mostly synchronization-related, which do objectively make implementing algorithms on UTF-8-encoded data simpler), "the number of set prefix bits is the number of bytes" doesn't seem meaningfully more complex than a single continuation bit.
  - Dylan16807 3y ago
    > "the number of set prefix bits is the number of bytes" Except when it's 1, because that's an invalid start, and except when it's 0, because that means the character is a single byte. And it also means you're dealing with three classes of byte now. Plus UTF-8 has more invalid encodings to deal with than a super-simple format.
    - morelisp 3y ago
      > Plus UTF-8 has more invalid encodings to deal with than a super-simple format. If your format supports non-canonical encodings you're in for a bad time no matter what, so a whole lot of that simplicity is fake. > And it also means you're dealing with three classes of byte now. If you're working a byte at a time you're doing it wrong, unless you're re-syncing an invalid stream in which case it's as simple as a continuation bit (specifically, it's two continuation bits).