5 ms·

Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: https://github.com/python/cpython/blob/master/Objects/unicodeobject.c h

Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: https://github.com/python/cpython/blob/master/Objects/unicodeobject.c https://github.com/python/cpython/blob/master/Objects/unicod...

the_mitsuhiko 11y ago
> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways. Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around. The object is too clever for its own good.
- klodolph 11y ago
  You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32). >>> import sys >>> def charsize(x): ... return sys.getsizeof(x + x) - sys.getsizeof(x) ... >>> charsize(b'A') 1 >>> charsize('A') 1 >>> charsize('\xff') 1 >>> charsize('\uffff') 2 >>> charsize('\U0010FFFF') 4
  - the_mitsuhiko 11y ago
    > But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
  - aidenn0 11y ago
    Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.
nostrademons 11y ago
Also, it's incompatible with UTF-8 strings stored in C, which means that when you cross the Python/C API boundary, you have to re-encode all strings. This is a large performance penalty right at the time when you can least afford performance penalties. IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.
- shoyer 11y ago
  This design doc from DyND (a possible NumPy alternative) has some useful references on this point: https://github.com/libdynd/libdynd/blob/master/docs/string-design.md#code-unit-api-not-code-point https://github.com/libdynd/libdynd/blob/master/docs/string-d...
  - rurban 11y ago
    Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit. Very common on every better VM. This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl. Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode. And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.
    - 0x0 11y ago
      "Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)
- viraptor 11y ago
  > Also, it's incompatible with UTF-8 strings stored in C That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/ https://www.python.org/dev/peps/pep-0393/ So unless you go outside of UCS-1, you don't need to reencode anything.
eugenekolo2 11y ago
Yeah, I'm not reading 15K LOC... What makes it clever? What is even the point of linking a 15K file?
Animats 11y ago
Another alternative would have been to store strings in UTF-8 and generate an array of indices to the starting point of each entity only if needed for random access. Subscripting a string would use the array of indices. Only a few standard member functions of string (find, rfind, index, rindex) return integer subscripts into a string. Those could return an opaque index object which has an index and a link to the string. Such objects could be used for string operations which need a string index. If an index object is forcibly converted to an integer, the array of indices has to be generated, but otherwise, it is unnecessary. You could even allow adding or subtracting integers from an index object, which would walk the UTF-8 string forwards or backwards as indicated. This would make UTF-8 strings usable with subscript and slice notation without exploding them to wide characters.

the_mitsuhiko 11y ago

> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading: Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways. Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around. The object is too clever for its own good.

klodolph 11y ago
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32). >>> import sys >>> def charsize(x): ... return sys.getsizeof(x + x) - sys.getsizeof(x) ... >>> charsize(b'A') 1 >>> charsize('A') 1 >>> charsize('\xff') 1 >>> charsize('\uffff') 2 >>> charsize('\U0010FFFF') 4
- the_mitsuhiko 11y ago
  > But there are space optimizations on the Python string object which make it so it doesn't waste that much space Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
- aidenn0 11y ago
  Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.

nostrademons 11y ago

Also, it's incompatible with UTF-8 strings stored in C, which means that when you cross the Python/C API boundary, you have to re-encode all strings. This is a large performance penalty right at the time when you can least afford performance penalties. IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.

shoyer 11y ago
This design doc from DyND (a possible NumPy alternative) has some useful references on this point: https://github.com/libdynd/libdynd/blob/master/docs/string-design.md#code-unit-api-not-code-point https://github.com/libdynd/libdynd/blob/master/docs/string-d...
- rurban 11y ago
  Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit. Very common on every better VM. This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl. Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode. And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.
  - 0x0 11y ago
    "Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)
viraptor 11y ago
> Also, it's incompatible with UTF-8 strings stored in C That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/ https://www.python.org/dev/peps/pep-0393/ So unless you go outside of UCS-1, you don't need to reencode anything.

eugenekolo2 11y ago

Yeah, I'm not reading 15K LOC... What makes it clever? What is even the point of linking a 15K file?

Animats 11y ago

Another alternative would have been to store strings in UTF-8 and generate an array of indices to the starting point of each entity only if needed for random access. Subscripting a string would use the array of indices. Only a few standard member functions of string (find, rfind, index, rindex) return integer subscripts into a string. Those could return an opaque index object which has an index and a link to the string. Such objects could be used for string operations which need a string index. If an index object is forcibly converted to an integer, the array of indices has to be generated, but otherwise, it is unnecessary. You could even allow adding or subtracting integers from an index object, which would walk the UTF-8 string forwards or backwards as indicated. This would make UTF-8 strings usable with subscript and slice notation without exploding them to wide characters.