3 ms·
Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
by saghm 6d ago
Unless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
- cyphar 6d agoEven better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width. Truly the worst of all worlds.
- mafuy 6d agoCorrect me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character. This would be because you can stack modifications as long as you like.
- ElectricalUnion 6d agoWhat you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.
- Dylan16807 6d agoThe first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.
- flohofwoe 5d agoYes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later. UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).