4 ms·

I'm confused because U+1F635 can be stored in 3 bytes? I see nothing above 0xFFFFFF which would need the 4th byte? Or am I being dense?

kangalioo 5y ago
In UTF-8, only some bits are used for the actual character codes. The rest are control bits for the decoder to know where a character starts and ends. https://en.m.wikipedia.org/wiki/UTF-8#Encoding https://en.m.wikipedia.org/wiki/UTF-8#Encoding
deleted 5y ago
[deleted]
chrismorgan 5y ago
U+1F635 is encoded in UTF-8 as bytes {f0, 9f, 98, b5}.
- kingcharles 5y ago
  Thank you, that explains it!
Dylan16807 5y ago
To give a slightly more complete answer as to why, utf-8 was designed to be both backwards compatible with ASCII and self-synchronizing. So you can't use half the values of a byte when encoding code points outside of ASCII, and the remaining 128 values need to be split between starting bytes and continuation bytes. At best you're going to fit about 6 bits into each byte, and unicode spans the equivalent of just over 20 bits, so you need 4 bytes max. utf-8 opts to keep encoding very simple, and in 2-4 bytes it can store 11, 16, and 21 bits respectively. If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. But at that point maybe you should be compressing your text instead. Or you could weaken the compatibility and create "utf-1 but better" to fit into 3 bytes.
- chrismorgan 5y ago
  > Actually it does seem feasible to make an encoding that's backwards compatible with all printable ASCII characters, and uses 3 bytes for all other code points. Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits. To represent all the rest: Unicode scalar values minus printable ASCII is U+0000–U+10FFFF minus U+D800–U+DFFF minus U+0020–U+007E: 1,111,969 possibilities, log₂(1111969) ≈ 20.085 bits of entropy. (Removing the 2048 surrogates and 95 printable ASCII characters barely made a dent, under 0.003 bits, but I had to remove it for informational correctness.) 95 printable ASCII characters, log₂(95) ≈ 6.57 bits. Subtract that reservation from the three bytes available and you’ve got about 17.43 bits left for the rest of Unicode, but you needed over 20. [Edit: I observe you’ve removed what I was quoting before I posted my comment. Still fun mathematics.] > If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.) —⁂— I just got nerd-sniped, didn’t I?
  - kingcharles 5y ago
    I had to Google ⁂ as I've never seen that character before. Fascinating! It makes a nice Christmas snowflake stand-in. I wonder what HN's rules are with sanitizing Unicode, as they won't allow emojis?
    - chrismorgan 5y ago
      I use “—⁂—” as my horizontal rule in plain text, via my Compose key (Compose+h+r). HN removes things that are officially marked as emoji, normalises whitespace to space (which annoys me because I sometimes want to type things like narrow no-break space in equations which it makes wider and loses the no-breakyness of), and that’s all I’ve noticed. Control characters probably get some treatment or other too.

kangalioo 5y ago

In UTF-8, only some bits are used for the actual character codes. The rest are control bits for the decoder to know where a character starts and ends. https://en.m.wikipedia.org/wiki/UTF-8#Encoding https://en.m.wikipedia.org/wiki/UTF-8#Encoding

deleted 5y ago

[deleted]

chrismorgan 5y ago

U+1F635 is encoded in UTF-8 as bytes {f0, 9f, 98, b5}.

kingcharles 5y ago
Thank you, that explains it!

Dylan16807 5y ago

To give a slightly more complete answer as to why, utf-8 was designed to be both backwards compatible with ASCII and self-synchronizing. So you can't use half the values of a byte when encoding code points outside of ASCII, and the remaining 128 values need to be split between starting bytes and continuation bytes. At best you're going to fit about 6 bits into each byte, and unicode spans the equivalent of just over 20 bits, so you need 4 bytes max. utf-8 opts to keep encoding very simple, and in 2-4 bytes it can store 11, 16, and 21 bits respectively. If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. But at that point maybe you should be compressing your text instead. Or you could weaken the compatibility and create "utf-1 but better" to fit into 3 bytes.

chrismorgan 5y ago
> Actually it does seem feasible to make an encoding that's backwards compatible with all printable ASCII characters, and uses 3 bytes for all other code points. Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits. To represent all the rest: Unicode scalar values minus printable ASCII is U+0000–U+10FFFF minus U+D800–U+DFFF minus U+0020–U+007E: 1,111,969 possibilities, log₂(1111969) ≈ 20.085 bits of entropy. (Removing the 2048 surrogates and 95 printable ASCII characters barely made a dent, under 0.003 bits, but I had to remove it for informational correctness.) 95 printable ASCII characters, log₂(95) ≈ 6.57 bits. Subtract that reservation from the three bytes available and you’ve got about 17.43 bits left for the rest of Unicode, but you needed over 20. [Edit: I observe you’ve removed what I was quoting before I posted my comment. Still fun mathematics.] > If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.) —⁂— I just got nerd-sniped, didn’t I?
- kingcharles 5y ago
  I had to Google ⁂ as I've never seen that character before. Fascinating! It makes a nice Christmas snowflake stand-in. I wonder what HN's rules are with sanitizing Unicode, as they won't allow emojis?
  - chrismorgan 5y ago
    I use “—⁂—” as my horizontal rule in plain text, via my Compose key (Compose+h+r). HN removes things that are officially marked as emoji, normalises whitespace to space (which annoys me because I sometimes want to type things like narrow no-break space in equations which it makes wider and loses the no-breakyness of), and that’s all I’ve noticed. Control characters probably get some treatment or other too.