3 ms·

In practice this isn't as big a problem as you're making it out to be, because UTF8 is compatible with ANSI. Nearly all of computing has made a similar transit

by dtech 5y ago

In practice this isn't as big a problem as you're making it out to be, because UTF8 is compatible with ANSI. Nearly all of computing has made a similar transition in the 2000's and 2010's, and I've certainly debugged my share of encoding issues, but in general it worked for a large majority of use cases. I think most people would've taken a few encoding issues over the 10+ years long migration.

chrismorgan 5y ago
If it were ASCII and UTF-8 that this was all about, it wouldn’t be as big a deal. But it’s not. Python 2 was a code pages world. Not an ASCII world. Also Python’s Unicode isn’t and hasn’t ever been UTF-8 anywhere; rather, it’s sequences of Unicode code points (note that that’s code points, not Unicode scalar values−this and the representation of str are a pair of things I find quite bafflingly bad about Python 3, because it means Python’s Unicode strings aren’t, y’know, Unicode strings, because they can be ill-formed, and they’re stuck with bad fixed-width representations so that any even vaguely interesting string will be stored as (ill-formed) UTF-32). Python 2’s str type was a mess, used for bytes data, for strings of uncertain encoding, and for Unicode strings. In a vacuum, you might say you could make that work by splitting it up (and indeed the types module was an early endeavour to do that sort of thing back before it became evident it was insufficient). The key is that this confusion and widespread misuse was rife through the standard library and other third-party libraries. If the Python 2 → 3 transition had been just str-as-bytes/str-as-who-knows-what/unicode → bytes/something/str, it might have been manageable. But the libraries were so shot that a straightforward migration was impossible, because str was too many things and it was quite impossible to tell statically what should be done about it. (And impossible dynamically to resolve as well.)
slavik81 5y ago
Python 3 IO isn't UTF-8. The entire point of the change was to support _arbitrary_ encodings (most of which are legacy code pages). Python would sometimes guess the output encoding on Windows to be ASCII even when UTF-8 was supported by the output device. So you'd end up with encoding-oblivious Python 2 programs that worked perfectly reading and writing UTF-8, and Python 3 programs that failed upon encountering a non-ASCII character. They've since improved the defaults on Windows, but the changes in Python 3 that caused me a lot of pain were when they made IO to be non-Unicode.
- kortex 5y ago
  Man, it's it's harrowing in hindsight to look back on porting 2->3 on linux, and remembering my coworkers that had to do it on Windows with encoding-oblivious libraries. They're still constantly finding bits of code that break cause some customer decided to switch from "delta" to “Δ” in some machine learning log output.

by dtech 5y ago

chrismorgan 5y ago

If it were ASCII and UTF-8 that this was all about, it wouldn’t be as big a deal. But it’s not. Python 2 was a code pages world. Not an ASCII world. Also Python’s Unicode isn’t and hasn’t ever been UTF-8 anywhere; rather, it’s sequences of Unicode code points (note that that’s code points, not Unicode scalar values−this and the representation of str are a pair of things I find quite bafflingly bad about Python 3, because it means Python’s Unicode strings aren’t, y’know, Unicode strings, because they can be ill-formed, and they’re stuck with bad fixed-width representations so that any even vaguely interesting string will be stored as (ill-formed) UTF-32). Python 2’s str type was a mess, used for bytes data, for strings of uncertain encoding, and for Unicode strings. In a vacuum, you might say you could make that work by splitting it up (and indeed the types module was an early endeavour to do that sort of thing back before it became evident it was insufficient). The key is that this confusion and widespread misuse was rife through the standard library and other third-party libraries. If the Python 2 → 3 transition had been just str-as-bytes/str-as-who-knows-what/unicode → bytes/something/str, it might have been manageable. But the libraries were so shot that a straightforward migration was impossible, because str was too many things and it was quite impossible to tell statically what should be done about it. (And impossible dynamically to resolve as well.)

slavik81 5y ago

Python 3 IO isn't UTF-8. The entire point of the change was to support _arbitrary_ encodings (most of which are legacy code pages). Python would sometimes guess the output encoding on Windows to be ASCII even when UTF-8 was supported by the output device. So you'd end up with encoding-oblivious Python 2 programs that worked perfectly reading and writing UTF-8, and Python 3 programs that failed upon encountering a non-ASCII character. They've since improved the defaults on Windows, but the changes in Python 3 that caused me a lot of pain were when they made IO to be non-Unicode.

kortex 5y ago
Man, it's it's harrowing in hindsight to look back on porting 2->3 on linux, and remembering my coworkers that had to do it on Windows with encoding-oblivious libraries. They're still constantly finding bits of code that break cause some customer decided to switch from "delta" to “Δ” in some machine learning log output.