3 ms·

Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working m

Do you understand unicode and it's encodings well enough to have an informed opinion that python3's braindamaged unicode implementation is superior to working mostly at the byte string level in python2? I do, and I do believe Python 3's approach is superior for strings. An array of 8-bit integers is not a string. It's an array of 8-bit integers. I want my string type to be a sequence of Unicode codepoints, and I want iterating it to yield codepoints, I want indexing into it to yield the codepoint at the index, and I want the length of it to be the number of codepoints it contains. When I'm ready to send it over a wire or flush it to disk, then I'll think about the best way to represent it as an array of 8-bit integers, but bringing the baggage of encodings (especially the baggage of UTF-8) into any part of my code except the boundary layer is asking for trouble.

patrec 9y ago
What's your use case for iterating or indexing into code points?
- ubernostrum 9y ago
  My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes. I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.
  - patrec 9y ago
    But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either a) low-level mucking around like normalization -- a fringe use b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.

patrec 9y ago

What's your use case for iterating or indexing into code points?

ubernostrum 9y ago
My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes. I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.
- patrec 9y ago
  But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either a) low-level mucking around like normalization -- a fringe use b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.