4 ms·

couldnt emoji fit in 3 bytes as well? I don't think there are that many ...

rakoo 5y ago
utf8 is an encoding for unicode codepoints. Those codepoints are spread on a space that is extremely vast (up to ~4 billion) for which that can be represented with up to 4 bytes. It turns out emojis are positioned in a place where the first byte will never be 0, so even if there were only one it would require the full 4 bytes to encode them.
- the_mitsuhiko 5y ago
  > Those codepoints are spread on a space that is extremely vast (up to ~4 billion) You are off by a lot. The maximum code point is about 21bit high (0x10FFFF). The space is only 1.1 million large.
  - capitainenemo 5y ago
    It could be he's thinking of the historical definition which included support for 5 character encoding with a maximum codepoint of U+7FFFFFFF or ~2 billion. https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History That was restricted I believe primarily for compatibility with more limited encodings like UTF-16. I guess it's possible that at some future point in human history when UTF-16 has been purged from memory, the 5 character encoding might be allowed again. :)
  - ninkendo 5y ago
    Related: time to explain UTF-8! (In case anybody is curious. I personally think it's extremely clever and worth understanding:) Characters <128 are encoded with a single byte: 0xxxxxxx Characters >128 are encoded with multiple bytes. A two-byte character looks like: 110xxxxx 10xxxxxx (11 useful bits, representing code points 128-2047) A three-byte character looks like: 1110xxxx 10xxxxxx 10xxxxxx (16 useful bits, representing code points 2048-65535) A four-byte character looks like: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (21 useful bits, representing code points 65536-2097151) Now, technically this scheme could expand to 6-byte characters without getting confused with things like BOM/etc, however any code points larger than 2^21 wouldn't be representable in UTF-16, which has its own set of constraints. This means the unicode consortium has basically limited themselves to two million or so possible code points, which is why UTF-8 doesn't need to go more than 4 bytes. (I wonder if a future unicode version will require a larger limit and would thus create a new "utf8mb6" scheme, and drop UTF-16 altogether?)
    - tialaramex 5y ago
      Unicode specifically limited itself to the range zero to U+10FFFF Obviously nothing in the laws of nature forbids "a future Unicode version" from disavowing this limit, but we could say the same for whether "a future United States of America" could disavow the status of independent Indian Tribes it has previously recognised.
    - cesarb 5y ago
      > (I wonder if a future unicode version will require a larger limit and would thus create a new "utf8mb6" scheme, and drop UTF-16 altogether?) On a thread a couple of years ago (https://news.ycombinator.com/item?id=20600873 https://news.ycombinator.com/item?id=20600873) it was mentioned that the UTF-8 encoding scheme can be cleanly extended to 36 bits, so even "utf8mb7" would be a possibility.
_moof 5y ago
Almost all emoji are four bytes in UTF-8.
- cblconfederate 5y ago
  were all the other positions taken?

utf8 is an encoding for unicode codepoints. Those codepoints are spread on a space that is extremely vast (up to ~4 billion) for which that can be represented with up to 4 bytes. It turns out emojis are positioned in a place where the first byte will never be 0, so even if there were only one it would require the full 4 bytes to encode them.

the_mitsuhiko 5y ago
> Those codepoints are spread on a space that is extremely vast (up to ~4 billion) You are off by a lot. The maximum code point is about 21bit high (0x10FFFF). The space is only 1.1 million large.
- capitainenemo 5y ago
  It could be he's thinking of the historical definition which included support for 5 character encoding with a maximum codepoint of U+7FFFFFFF or ~2 billion. https://en.wikipedia.org/wiki/UTF-8#History https://en.wikipedia.org/wiki/UTF-8#History That was restricted I believe primarily for compatibility with more limited encodings like UTF-16. I guess it's possible that at some future point in human history when UTF-16 has been purged from memory, the 5 character encoding might be allowed again. :)
- ninkendo 5y ago
  Related: time to explain UTF-8! (In case anybody is curious. I personally think it's extremely clever and worth understanding:) Characters <128 are encoded with a single byte: 0xxxxxxx Characters >128 are encoded with multiple bytes. A two-byte character looks like: 110xxxxx 10xxxxxx (11 useful bits, representing code points 128-2047) A three-byte character looks like: 1110xxxx 10xxxxxx 10xxxxxx (16 useful bits, representing code points 2048-65535) A four-byte character looks like: 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (21 useful bits, representing code points 65536-2097151) Now, technically this scheme could expand to 6-byte characters without getting confused with things like BOM/etc, however any code points larger than 2^21 wouldn't be representable in UTF-16, which has its own set of constraints. This means the unicode consortium has basically limited themselves to two million or so possible code points, which is why UTF-8 doesn't need to go more than 4 bytes. (I wonder if a future unicode version will require a larger limit and would thus create a new "utf8mb6" scheme, and drop UTF-16 altogether?)
  - tialaramex 5y ago
    Unicode specifically limited itself to the range zero to U+10FFFF Obviously nothing in the laws of nature forbids "a future Unicode version" from disavowing this limit, but we could say the same for whether "a future United States of America" could disavow the status of independent Indian Tribes it has previously recognised.
  - cesarb 5y ago
    > (I wonder if a future unicode version will require a larger limit and would thus create a new "utf8mb6" scheme, and drop UTF-16 altogether?) On a thread a couple of years ago (https://news.ycombinator.com/item?id=20600873 https://news.ycombinator.com/item?id=20600873) it was mentioned that the UTF-8 encoding scheme can be cleanly extended to 36 bits, so even "utf8mb7" would be a possibility.

_moof 5y ago

Almost all emoji are four bytes in UTF-8.

cblconfederate 5y ago
were all the other positions taken?