3 ms·

> Python 3's unicode model is completely outdated but was supposed to be the big feature of the version. While that was happening however the rest of the world

by avian 8y ago

> Python 3's unicode model is completely outdated but was supposed to be the big feature of the version. While that was happening however the rest of the world started moving over to UTF-8. I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently? I’ve worked on Perl that uses that approach and it’s an absolute nightmare. All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely. You get hard-to-find bugs when some code somewhere introduces invalid UTF to a string. The only real benefit you get is slightly smaller memory footprint when dealing mostly with English text. I much prefer Python’s decode-process-encode approach.

rurban 8y ago
The python nightmare is purely performance due to its use of the UCS-2 or 4 encoding (wchar_t without surrogate pairs). UTF-8 was ruled out for its inability for direct indexing, but this proved wrong. 4x longer string buffers trash the cache much more than the needed CPU cycles for utf-8 decoding. perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else. so the context needs to store it, like IO, process. and for performance they silently loose the utf-8 flag for 128—255 codepoints. but all this is still better than the libc/POSIX design failure to use ENV vars, where the encoding is global, not localized. every single string API needs to do a getenv, and its interpretation is action at a distance.
- avian 8y ago
  > it's only latin-1 or utf-8 There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.
  - rurban 8y ago
    No. Foreign encodings are internally converted to utf-8 via the Encoding library and IO layers. The string library is only done for latin-1 and utf-8. The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.
the_mitsuhiko 8y ago
> I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently? Correct. Because the current approach completely fell apart when people started using emojis. The internal encoding in too many cases is now always UCS4 which is very wasteful. > All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely. Go, Rust and Swift all show that using UTF-8 internally is not hard. Not sure why you think its hard. I have written a lot of code in those languages now and I feel much happier with their unicode support than Python's.

by avian 8y ago

rurban 8y ago

The python nightmare is purely performance due to its use of the UCS-2 or 4 encoding (wchar_t without surrogate pairs). UTF-8 was ruled out for its inability for direct indexing, but this proved wrong. 4x longer string buffers trash the cache much more than the needed CPU cycles for utf-8 decoding. perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else. so the context needs to store it, like IO, process. and for performance they silently loose the utf-8 flag for 128—255 codepoints. but all this is still better than the libc/POSIX design failure to use ENV vars, where the encoding is global, not localized. every single string API needs to do a getenv, and its interpretation is action at a distance.

avian 8y ago
> it's only latin-1 or utf-8 There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.
- rurban 8y ago
  No. Foreign encodings are internally converted to utf-8 via the Encoding library and IO layers. The string library is only done for latin-1 and utf-8. The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.

the_mitsuhiko 8y ago

> I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently? Correct. Because the current approach completely fell apart when people started using emojis. The internal encoding in too many cases is now always UCS4 which is very wasteful. > All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely. Go, Rust and Swift all show that using UTF-8 internally is not hard. Not sure why you think its hard. I have written a lot of code in those languages now and I feel much happier with their unicode support than Python's.