3 ms·
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste
by klodolph 11y ago
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32).
>>> import sys
>>> def charsize(x):
... return sys.getsizeof(x + x) - sys.getsizeof(x)
...
>>> charsize(b'A')
1
>>> charsize('A')
1
>>> charsize('\xff')
1
>>> charsize('\uffff')
2
>>> charsize('\U0010FFFF')
4
- the_mitsuhiko 11y ago> But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
- aidenn0 11y agoExcept UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.