4 ms·

Tom is why I understand UTF-8 [1]. And it is not so much WHAT he explained but HOW he explained it, what really made it stick. It was his sheer unbridled genui

Tom is why I understand UTF-8 [1]. And it is not so much WHAT he explained but HOW he explained it, what really made it stick. It was his sheer unbridled genuine enthusiasm that put him on the map for me. One of the best to do it. [1] https://www.youtube.com/watch?v=MijmeoH9LT4 https://www.youtube.com/watch?v=MijmeoH9LT4

devoutsalsa 3y ago
And he's why I know I shouldn't roll my own time zone library. https://youtu.be/-5wpm-gesOY?si=eL38cruwJZDiuHVS https://youtu.be/-5wpm-gesOY?si=eL38cruwJZDiuHVS
chubot 3y ago
This is a great video! I’ve been writing about UTF-8 on my blog and I noticed that many programmers don’t understand it, after 10 or 20 years The main impediment seems to be that languages like Java, JavaScript, and Python treat UTF-8 as just another encoding, but really it’s the most fundamental encoding. The language abstractions get in the way and distort the way people think about text Newer languages like Go and Rust are more sensible, they don’t have global mutable encoding variables
- omsimun 3y ago
  It's misleading to describe UTF-8 as "the most fundamental encoding", because of the existence of UTF-32 (essentially just a trivial encoding of "raw" Unicode code points) and UTF-16 (which has certain limitations that later became part of the specification of what code points are valid in any encoding) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surrogates) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surro... UTF-8 is the most ubiquitous encoding on the web, but that doesn't make it more fundamental than any other.
  - lifthrasiir 3y ago
    The word "fundamental" here probably has the same meaning as "elementary" as in elementary mathematics, i.e. something you should absolutely understand first, not something that makes up everything else.
- jfengel 3y ago
  What's not to understand about UTF8? It's a way of coding up points in the Unicode space. You decode the bits and it yields a number that corresponds to a glyph, possibly with some decorations. The only thing special about UTF8 is that it happens to write ASCII as ASCII, which is nice as long as you realize that there is much outside that. Either that, or I'm completely off base and part of the vast horde who don't get it.
  - henriquez 3y ago
    I read your comment in the “Simpson comic book nerd” tone of voice.
  - Dylan16807 3y ago
    Variable width encodings are inherently harder to understand. And while UTF-8 tries very hard to be simple whenever possible, it does have some nonobvious constraints that make it significantly more complex than the actual simplest variable-width encoding (a continuation bit and then 7 data bits).
    - morelisp 3y ago
      Even ignoring all the other advantages (mostly synchronization-related, which do objectively make implementing algorithms on UTF-8-encoded data simpler), "the number of set prefix bits is the number of bytes" doesn't seem meaningfully more complex than a single continuation bit.
      - Dylan16807 3y ago
        > "the number of set prefix bits is the number of bytes" Except when it's 1, because that's an invalid start, and except when it's 0, because that means the character is a single byte. And it also means you're dealing with three classes of byte now. Plus UTF-8 has more invalid encodings to deal with than a super-simple format.
        morelisp 3y ago
Nican 3y ago
I first came to learn about the complexity of character sets by finding out that SQL Server's default characterset is 2 bytes per character. I eventually came across Scott's video. UTF-8 encoding is only as new as SQL Server 2019. https://sqlquantumleap.com/2018/09/28/native-utf-8-support-in-sql-server-2019-savior-false-prophet-or-both/ https://sqlquantumleap.com/2018/09/28/native-utf-8-support-i...
mixmastamyk 3y ago
Interesting… never heard of the “null problem” as a requirement before.
palemoonale 3y ago
Half way in, i find this a tiring style of narration. Less enthusiasm would require less focus and thus less energy from the viewer.

devoutsalsa 3y ago

And he's why I know I shouldn't roll my own time zone library. https://youtu.be/-5wpm-gesOY?si=eL38cruwJZDiuHVS https://youtu.be/-5wpm-gesOY?si=eL38cruwJZDiuHVS

chubot 3y ago

This is a great video! I’ve been writing about UTF-8 on my blog and I noticed that many programmers don’t understand it, after 10 or 20 years The main impediment seems to be that languages like Java, JavaScript, and Python treat UTF-8 as just another encoding, but really it’s the most fundamental encoding. The language abstractions get in the way and distort the way people think about text Newer languages like Go and Rust are more sensible, they don’t have global mutable encoding variables

omsimun 3y ago
It's misleading to describe UTF-8 as "the most fundamental encoding", because of the existence of UTF-32 (essentially just a trivial encoding of "raw" Unicode code points) and UTF-16 (which has certain limitations that later became part of the specification of what code points are valid in any encoding) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surrogates) https://en.wikipedia.org/wiki/UTF-16#U+D800_to_U+DFFF_(surro... UTF-8 is the most ubiquitous encoding on the web, but that doesn't make it more fundamental than any other.
- lifthrasiir 3y ago
  The word "fundamental" here probably has the same meaning as "elementary" as in elementary mathematics, i.e. something you should absolutely understand first, not something that makes up everything else.
jfengel 3y ago
What's not to understand about UTF8? It's a way of coding up points in the Unicode space. You decode the bits and it yields a number that corresponds to a glyph, possibly with some decorations. The only thing special about UTF8 is that it happens to write ASCII as ASCII, which is nice as long as you realize that there is much outside that. Either that, or I'm completely off base and part of the vast horde who don't get it.
- henriquez 3y ago
  I read your comment in the “Simpson comic book nerd” tone of voice.
- Dylan16807 3y ago
  Variable width encodings are inherently harder to understand. And while UTF-8 tries very hard to be simple whenever possible, it does have some nonobvious constraints that make it significantly more complex than the actual simplest variable-width encoding (a continuation bit and then 7 data bits).
  - morelisp 3y ago
    Even ignoring all the other advantages (mostly synchronization-related, which do objectively make implementing algorithms on UTF-8-encoded data simpler), "the number of set prefix bits is the number of bytes" doesn't seem meaningfully more complex than a single continuation bit.
    - Dylan16807 3y ago
      > "the number of set prefix bits is the number of bytes" Except when it's 1, because that's an invalid start, and except when it's 0, because that means the character is a single byte. And it also means you're dealing with three classes of byte now. Plus UTF-8 has more invalid encodings to deal with than a super-simple format.
      - morelisp 3y ago

Nican 3y ago

I first came to learn about the complexity of character sets by finding out that SQL Server's default characterset is 2 bytes per character. I eventually came across Scott's video. UTF-8 encoding is only as new as SQL Server 2019. https://sqlquantumleap.com/2018/09/28/native-utf-8-support-in-sql-server-2019-savior-false-prophet-or-both/ https://sqlquantumleap.com/2018/09/28/native-utf-8-support-i...

mixmastamyk 3y ago

Interesting… never heard of the “null problem” as a requirement before.

palemoonale 3y ago

Half way in, i find this a tiring style of narration. Less enthusiasm would require less focus and thus less energy from the viewer.