3 ms·
It optimized for O(1) indexing to Unicode scalar values instead of optimizing for more useful things. Also, even though the API semantics are UTF-32, the storag
by hsivonen 9y ago
It optimized for O(1) indexing to Unicode scalar values instead of optimizing for more useful things. Also, even though the API semantics are UTF-32, the storage may be USC-2 or Latin-1, so there's a surprise performance effect when a string gets it first non-Latin-1 character or its first non-BMP character. So much for O(1) behavior.
The best in-memory representation of Unicode is UTF-8. No alignment issues, ingest "decode" is mere validation and output "encode" step is a no-op. Space requirements are favorable to common computer language syntaxes. Byte-wise lexical sort matches code point-wise lexical sort.
However, since "narrow builds" of Python 2 already used UTF-16 which allowed for things like bridging strings with Cocoa, Python would have done well to stick to UTF-16 and get rid of the "wide builds".
Not having both "narrow" (UTF-16, Python 2 on Mac) and "wide" (UTF-32, Python 2 on Debian) is an improvement, though.
- jwilk 9y agoPython strings are immutable. The only time a string can get its first non-Latin-1 character is at construction time, which is necessarily O(n).