3 ms·

Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a

Exactly this. People conflate unicode with encoding quite a bit. I think it was plan9 and early Go that used "runes" as a unit, where one or more runes formed a character and an array of runes could be encoded into bytes using a given encoding. The in memory size of a rune was just an implementation detail, and while it could be important for the programmer that the size of a rune was 2 bytes, this didn't mean the length of an array of 2 runes was 4. I always liked the rune unit, and while my memory is hazy I think it was just code points. I think part of the issue is programmers and apis mixing bit units for in memory representation of a conceptual value mapping (unicode), conceptual characters, stored size when encoded and so on ... without firming up those abstractions with interfaces. It gets lossy.

eesmith 3y ago
FWIW, CPython uses one of several Unicode string implementation representations, depending on the code points involved: >>> import sys >>> s = 'A' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 1049 >>> s = '\N{SNOWMAN WITHOUT SNOW}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 2074 >>> s = '\N{MUSICAL SYMBOL G CLEF}' * 1000 >>> len(s) 1000 >>> sys.getsizeof(s) 4076 See https://peps.python.org/pep-0393/ https://peps.python.org/pep-0393/ . Mentioned in the linked-to article with "CPython since 3.3 makes the same idea three-level with code point semantics".