3 ms·

Python 3 has some architectural decisions that make working with data difficult; mostly ///my opinion/// is that it completely botched Unicode handling. The id

Python 3 has some architectural decisions that make working with data difficult; mostly ///my opinion/// is that it completely botched Unicode handling. The idea of data being encoded should be an extremely high level idea, with foundations that are supported by the internal libraries and maybe even compiler. I feel that golang has a much more data-structure friendly view of Unicode. * The ancient paradigm of bytes remains * Strings are immutable sequences of bytes; but are trivial to cast back and forth. * 'runes' (distinct Unicode code points) also exist However to actually validate or do anything fancy to a Unicode sequence a programmer would need libraries from https://godoc.org/golang.org/x/text https://godoc.org/golang.org/x/text which shows how truly complex /proper/ handling of Unicode data is; instead of just the simple copy / append / search (is this inside of / a match to it) operations that make up 99% of my handling of Unicode data.

Siecje 9y ago
What about Python3's unicode handling is botched?
- hsivonen 9y ago
  It optimized for O(1) indexing to Unicode scalar values instead of optimizing for more useful things. Also, even though the API semantics are UTF-32, the storage may be USC-2 or Latin-1, so there's a surprise performance effect when a string gets it first non-Latin-1 character or its first non-BMP character. So much for O(1) behavior. The best in-memory representation of Unicode is UTF-8. No alignment issues, ingest "decode" is mere validation and output "encode" step is a no-op. Space requirements are favorable to common computer language syntaxes. Byte-wise lexical sort matches code point-wise lexical sort. However, since "narrow builds" of Python 2 already used UTF-16 which allowed for things like bridging strings with Cocoa, Python would have done well to stick to UTF-16 and get rid of the "wide builds". Not having both "narrow" (UTF-16, Python 2 on Mac) and "wide" (UTF-32, Python 2 on Debian) is an improvement, though.
  - jwilk 9y ago
    Python strings are immutable. The only time a string can get its first non-Latin-1 character is at construction time, which is necessarily O(n).
flossball 9y ago
Have you worked in Python 2 unicode?? Over half of the work and debugging the search engine service management team I was on was fixing unicode document injesting. But I agree, making unicode part of the language is very dumb. I just wish they would give up as it has never been good.

Siecje 9y ago

What about Python3's unicode handling is botched?

hsivonen 9y ago
It optimized for O(1) indexing to Unicode scalar values instead of optimizing for more useful things. Also, even though the API semantics are UTF-32, the storage may be USC-2 or Latin-1, so there's a surprise performance effect when a string gets it first non-Latin-1 character or its first non-BMP character. So much for O(1) behavior. The best in-memory representation of Unicode is UTF-8. No alignment issues, ingest "decode" is mere validation and output "encode" step is a no-op. Space requirements are favorable to common computer language syntaxes. Byte-wise lexical sort matches code point-wise lexical sort. However, since "narrow builds" of Python 2 already used UTF-16 which allowed for things like bridging strings with Cocoa, Python would have done well to stick to UTF-16 and get rid of the "wide builds". Not having both "narrow" (UTF-16, Python 2 on Mac) and "wide" (UTF-32, Python 2 on Debian) is an improvement, though.
- jwilk 9y ago
  Python strings are immutable. The only time a string can get its first non-Latin-1 character is at construction time, which is necessarily O(n).

flossball 9y ago

Have you worked in Python 2 unicode?? Over half of the work and debugging the search engine service management team I was on was fixing unicode document injesting. But I agree, making unicode part of the language is very dumb. I just wish they would give up as it has never been good.