4 ms·
Python is UTF-8 by default and only upgrades to UTF-16 / 32 when it would make sense to do so given the characters in the string. > UTF-32 is wrong, [because]
by svieira 11y ago
Python is UTF-8 by default and only upgrades to UTF-16 / 32 when it would make sense to do so given the characters in the string.
> UTF-32 is wrong, [because] it's wasteful and still doesn't accomplish what people naively expect due to grapheme clusters potentially taking more than one UTF-32 code unit.
Out of curiosity, is there a correct way to encode unicode that doesn't involve this level of surprise? I thought that this was still an unsolved problem at this point.
- Dylan16807 11y agoYou get people to accept the truth that characters have a variable length in bytes. Then you offer a data structure that lets you perform O(1) or O(logn) operations on sequences of single-character strings. If it's read-only you could make it just be an index, blah blah the details don't matter a lot, the point is you can make something that's both correct to grapheme clusters and probably more space-efficient than UTF-32 despite the extra data. And then the encoding inside the character strings isn't particularly important, but might as well use UTF-8. - Either that or make yourself a hilariously inefficient format based on: UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in Stream-Safe Text Format if it would not contain any sequences of non-starters longer than 30 characters in length when normalized to NFKD. Who's with me on 128-byte characters.