4 ms·

I looked through the master emoji list and I can't see that any expand into the 4th byte, but this stuff is REALLY complicated and I have no idea. https://unic

I looked through the master emoji list and I can't see that any expand into the 4th byte, but this stuff is REALLY complicated and I have no idea. https://unicode.org/Public/emoji/14.0/emoji-test.txt https://unicode.org/Public/emoji/14.0/emoji-test.txt

chrismorgan 5y ago
I believe you’re confusing two distinct concepts: Sounds like you’re looking at how many scalar values are in an extended grapheme cluster (e.g. U+1F635 U+200D U+1F4AB dizzy face + zero width joiner + dizzy symbol, that’s one “character” made up of three scalar values). The thing in question here is UTF-8 code units (bytes), and how many UTF-8 needs to encode a single scalar value. UTF-8 needs four bytes for anything above U+FFFF, and almost all emoji are above that.
- kingcharles 5y ago
  I'm confused because U+1F635 can be stored in 3 bytes? I see nothing above 0xFFFFFF which would need the 4th byte? Or am I being dense?
  - kangalioo 5y ago
    In UTF-8, only some bits are used for the actual character codes. The rest are control bits for the decoder to know where a character starts and ends. https://en.m.wikipedia.org/wiki/UTF-8#Encoding https://en.m.wikipedia.org/wiki/UTF-8#Encoding
  - deleted 5y ago
    [deleted]
  - chrismorgan 5y ago
    U+1F635 is encoded in UTF-8 as bytes {f0, 9f, 98, b5}.
    - kingcharles 5y ago
      Thank you, that explains it!
  - Dylan16807 5y ago
    To give a slightly more complete answer as to why, utf-8 was designed to be both backwards compatible with ASCII and self-synchronizing. So you can't use half the values of a byte when encoding code points outside of ASCII, and the remaining 128 values need to be split between starting bytes and continuation bytes. At best you're going to fit about 6 bits into each byte, and unicode spans the equivalent of just over 20 bits, so you need 4 bytes max. utf-8 opts to keep encoding very simple, and in 2-4 bytes it can store 11, 16, and 21 bits respectively. If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. But at that point maybe you should be compressing your text instead. Or you could weaken the compatibility and create "utf-1 but better" to fit into 3 bytes.
    - chrismorgan 5y ago
      > Actually it does seem feasible to make an encoding that's backwards compatible with all printable ASCII characters, and uses 3 bytes for all other code points. Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits. To represent all the rest: Unicode scalar values minus printable ASCII is U+0000–U+10FFFF minus U+D800–U+DFFF minus U+0020–U+007E: 1,111,969 possibilities, log₂(1111969) ≈ 20.085 bits of entropy. (Removing the 2048 surrogates and 95 printable ASCII characters barely made a dent, under 0.003 bits, but I had to remove it for informational correctness.) 95 printable ASCII characters, log₂(95) ≈ 6.57 bits. Subtract that reservation from the three bytes available and you’ve got about 17.43 bits left for the rest of Unicode, but you needed over 20. [Edit: I observe you’ve removed what I was quoting before I posted my comment. Still fun mathematics.] > If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.) —⁂— I just got nerd-sniped, didn’t I?
- tomwojcik 5y ago
  I've been struggling with fixing some edge cases with emojis in latin1 encoded DB this past couple of days and what you mention here is the most important thing I found yesterday (conincidence). Unicode strings are made up of Unicode code points, but some code points combine with others to make graphemes (single visual characters). Some emojis, like flags (U+1F1EA U+1F1E6 ), are made of two code points. EDIT: Flag unicode.
  - chrismorgan 5y ago
    Correction: Unicode strings are made up of scalar values, not code points. The difference is that accursed mess that UTF-16 foist upon us, surrogates;oU+D800–U+DFFF are code points, but are not valid scalar values and cannot occur in Unicode strings (even if things like JavaScript are stupid in allowing potentially-malformed UTF-16, which is incompatible with Unicode, and Python similarly stupid in having strings that are not Unicode but are instead sequences of code points).
    - tomwojcik 5y ago
      Thanks. I think I will need to read a proper book / article on this subject as encoding in general keeps haunting me periodically.

chrismorgan 5y ago

I believe you’re confusing two distinct concepts: Sounds like you’re looking at how many scalar values are in an extended grapheme cluster (e.g. U+1F635 U+200D U+1F4AB dizzy face + zero width joiner + dizzy symbol, that’s one “character” made up of three scalar values). The thing in question here is UTF-8 code units (bytes), and how many UTF-8 needs to encode a single scalar value. UTF-8 needs four bytes for anything above U+FFFF, and almost all emoji are above that.

kingcharles 5y ago
I'm confused because U+1F635 can be stored in 3 bytes? I see nothing above 0xFFFFFF which would need the 4th byte? Or am I being dense?
- kangalioo 5y ago
  In UTF-8, only some bits are used for the actual character codes. The rest are control bits for the decoder to know where a character starts and ends. https://en.m.wikipedia.org/wiki/UTF-8#Encoding https://en.m.wikipedia.org/wiki/UTF-8#Encoding
- deleted 5y ago
  [deleted]
- chrismorgan 5y ago
  U+1F635 is encoded in UTF-8 as bytes {f0, 9f, 98, b5}.
  - kingcharles 5y ago
    Thank you, that explains it!
- Dylan16807 5y ago
  To give a slightly more complete answer as to why, utf-8 was designed to be both backwards compatible with ASCII and self-synchronizing. So you can't use half the values of a byte when encoding code points outside of ASCII, and the remaining 128 values need to be split between starting bytes and continuation bytes. At best you're going to fit about 6 bits into each byte, and unicode spans the equivalent of just over 20 bits, so you need 4 bytes max. utf-8 opts to keep encoding very simple, and in 2-4 bytes it can store 11, 16, and 21 bits respectively. If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. But at that point maybe you should be compressing your text instead. Or you could weaken the compatibility and create "utf-1 but better" to fit into 3 bytes.
  - chrismorgan 5y ago
    > Actually it does seem feasible to make an encoding that's backwards compatible with all printable ASCII characters, and uses 3 bytes for all other code points. Sorry, you can’t squeeze Unicode in that way, even unsynchronised: you fall just over 2½ bits short. No, reserving the 95 printable ASCII characters scuttles your hopes, meaning you get less than 1½ bits of that byte to use for the rest of Unicode, when you needed a touch over 4 bits. To represent all the rest: Unicode scalar values minus printable ASCII is U+0000–U+10FFFF minus U+D800–U+DFFF minus U+0020–U+007E: 1,111,969 possibilities, log₂(1111969) ≈ 20.085 bits of entropy. (Removing the 2048 surrogates and 95 printable ASCII characters barely made a dent, under 0.003 bits, but I had to remove it for informational correctness.) 95 printable ASCII characters, log₂(95) ≈ 6.57 bits. Subtract that reservation from the three bytes available and you’ve got about 17.43 bits left for the rest of Unicode, but you needed over 20. [Edit: I observe you’ve removed what I was quoting before I posted my comment. Still fun mathematics.] > If you discarded ASCII compatibility, you could create a 1-3 byte self-synchronizing unicode encoding. Not quite. If you discarded ASCII compatibility, then you’d have almost four bits to spare, which is enough for a self-synchronising fixed-width 3-byte encoding (reserving the first bit of each byte to signal if it’s a start byte) or for an unsynchronised variable-width encoding of some kind; but to make it even 2-or-3 or 1-or-3 (rather than 1–3) bytes wide variable-width, you’d need four bits (3 + log₂ 2) for self-synchronisation, and you’re about 0.085 bits short. (1–3 bytes variable-width would require 3 + log₂ 3 ≈ 4.58 bits.) —⁂— I just got nerd-sniped, didn’t I?
tomwojcik 5y ago
I've been struggling with fixing some edge cases with emojis in latin1 encoded DB this past couple of days and what you mention here is the most important thing I found yesterday (conincidence). Unicode strings are made up of Unicode code points, but some code points combine with others to make graphemes (single visual characters). Some emojis, like flags (U+1F1EA U+1F1E6 ), are made of two code points. EDIT: Flag unicode.
- chrismorgan 5y ago
  Correction: Unicode strings are made up of scalar values, not code points. The difference is that accursed mess that UTF-16 foist upon us, surrogates;oU+D800–U+DFFF are code points, but are not valid scalar values and cannot occur in Unicode strings (even if things like JavaScript are stupid in allowing potentially-malformed UTF-16, which is incompatible with Unicode, and Python similarly stupid in having strings that are not Unicode but are instead sequences of code points).
  - tomwojcik 5y ago
    Thanks. I think I will need to read a proper book / article on this subject as encoding in general keeps haunting me periodically.