3 ms·
OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
by flohofwoe 13d ago
OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).
- DmitryOlshansky 13d agoThe bonus is synchonizing at arbitrary point in stream and that ASCII is UTF-8
- saghm 13d agoUnless I'm misremembering, even UTF-16 is variable. You need to bump up to UTF-32 to get fixed-width.
- cyphar 13d agoEven better, it's arguably both -- surrogate characters are valid codepoint values so technically UTF-16 is fixed-width but programs need to have special handling for surrogate pairs meaning it is practically variable-width. Truly the worst of all worlds.
- mafuy 13d agoCorrect me if I'm wrong, but I think all kinds of UTF, including 16 and 32, support arbitrary length for a single effective character. This would be because you can stack modifications as long as you like.
- ElectricalUnion 13d agoWhat you meant by "single effective character" is grapheme clusters. This whole discussion is about variable sized code points.
- Dylan16807 13d agoThe first comment was kind of iffy when it was also talking about buffers and characters, and focusing on code points is mostly a bad focus. It's worth bringing up so nobody thinks fixed width at a single layer is particularly useful, because other layers will still be variable.
- flohofwoe 12d agoYes, UTF-16 is the worst of all alternatives and should be abolished rather sooner than later. UTF-32 is fixed-width for UNICODE code points, but a single visual character (e.g. a "grapheme cluster") can be built from multiple code points. This is separate from the encoding algorithm though, grapheme clusters are mostly a problem for the high level code working with already decoded text data (text rendering, comparison, sorting etc...).