4 ms·

> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: Clever is relative. The Python 3 unicode string objects makes very l

> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways. Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around. The object is too clever for its own good.

klodolph 11y ago
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32). >>> import sys >>> def charsize(x): ... return sys.getsizeof(x + x) - sys.getsizeof(x) ... >>> charsize(b'A') 1 >>> charsize('A') 1 >>> charsize('\xff') 1 >>> charsize('\uffff') 2 >>> charsize('\U0010FFFF') 4
- the_mitsuhiko 11y ago
  > But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
- aidenn0 11y ago
  Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.

klodolph 11y ago

You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32). >>> import sys >>> def charsize(x): ... return sys.getsizeof(x + x) - sys.getsizeof(x) ... >>> charsize(b'A') 1 >>> charsize('A') 1 >>> charsize('\xff') 1 >>> charsize('\uffff') 2 >>> charsize('\U0010FFFF') 4

the_mitsuhiko 11y ago
> But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
aidenn0 11y ago
Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.