4 ms·
Technical errors regarding how UTF encodings work: * UTF-8 is at maximum four bytes per codepoint (which aren't characters, since multiple codepoints can make
by chungy 8y ago
Technical errors regarding how UTF encodings work:
* UTF-8 is at maximum four bytes per codepoint (which aren't characters, since multiple codepoints can make up a single character... see combining diaretics or ZWJ emoji things). "additional 1-4 bytes" should be "additional 1-3 bytes" (Historically, UTF-8 did support 5- and 6-byte sequences, and if you read up on it, the implementation is obvious, but that was before Unicode settled on U+10FFFF[1] being the maximal codepoint for backwards compatibility reasons with UCS-2...)
* It says UTF-16 extends UCS-2 (which is correct) by incorporating a 4-bit encoding... which probably was intended to be a "4-byte encoding" instead. The surrogate pairs reserve their first six bits for an additional 10 bits making up half of the codepoint being stored (whose value is later added to by 0x10000 to get the actual codepoint). Also, it provides 16 additional planes to the BMP. 17 total including the BMP.
[1] Actually, U+10FFFD is the maximum legal codepoint, the last two codepoints on every plane are reserved and declared as noncharacters. Few parsers are really so strict.
- jwilk 8y ago> Actually, U+10FFFD is the maximum legal codepoint, the last two codepoints on every plane are reserved and declared as noncharacters. Few parsers are really so strict. This is what Unicode 11.0, §2.4 says: Noncharacter code points are reserved for internal use, such as for sentinel values. They have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange. It would wrong for a general-purpose parser to reject these codepoints.
- chungy 8y agoHmm, alright. I have seen parsers reject them, presumably from misinterpreting the standard. Thanks for the correction :)
- bitcrazed 8y agoHi - Windows Console PM and author of the post here :) You're absolutely correct.I made some fixes, adjustments, and clarifications to the post this afternoon - hope they're a little more accurate. We'll be digging into this area in much more detail in future posts.