2 ms·
I think his reference to Unicode code points was actually based on UTF-16/8 mismatches, as in "Don't assume one character == one 16-bit char", since the upper c
by michaelochurch 15y ago
I think his reference to Unicode code points was actually based on UTF-16/8 mismatches, as in "Don't assume one character == one 16-bit char", since the upper code points (U+10000 and up) actually use two chars in, e.g., Java's UTF-16 representation.
There's a lot of code out there that works fine with one-char code points but breaks on two-char code points.
- derleth 15y agoKeep in mind UTF-8 can represent everything in Unicode without the surrogate pairs that make UTF-16 as annoying as it is. UTF-8 is inherently variable-width, but it's a flat scheme in that one universal mechanism suffices to represent everything you can represent. UTF-16 appears fixed-width until you realize it isn't, at which point you discover it's segmented and the segmentation scheme is both somewhat complex and likely to be missed entirely in naïve testing.
- SomeCallMeTim 15y agoI don't think that's what he meant. A surrogate pair in UTF-16 is still referring to a single code point. It's a good point that it's a place where people could screw up in dealing with Unicode, but it doesn't match his complaint. What does match is the fact that you can use a huge number of combining characters [1] to form a single glyph; each combining character and the base are a code point, so in order to figure out how many glyphs there are you have to iterate with the knowledge of what code points are combining characters. [1] https://en.wikipedia.org/wiki/Combining_character https://en.wikipedia.org/wiki/Combining_character