4 ms·

It didn't need to be such a disruptive breaking change that fragmented the ecosystem. They should have made the unicode handling changes opt-in per package or

• im3w1l 5y ago
  It's well known that you should make the "correct" choice the default. So unicode should have been opt-out (from __past__ import legacy_strings). Result: Everyone batch-adds unicode opt-out to their whole codebase, but new files use the modern way. Gradually people remove the opt outs as they go over old code.
  • TheCoelacanth 5y ago
    The correct way should be the default, but in the case where you already chose the wrong default, you shouldn't introduce the correct way of handling things and make it the default all at once. Upgrading to a new major version should never require changes to your code if you fixed all deprecation warnings in the previous major version. Otherwise, you make incremental upgrades impossible. The new alternative should always be introduced as opt-in with the old default deprecated, and then made default in the next major version.
  • pwdisswordfish8 5y ago
    This couldn’t work. For starters, the names of classes, attributes and functions are str in both versions, which is to say, bytestrings in Python 2 and Unicode strings in Python 3. This cannot be handwaved away with a pragma.
    • im3w1l 5y ago
      str = int is perfectly valid python. So having str mean different things in different modules is already possible.
      • pwdisswordfish8 5y ago
        You missed the point. It’s not about names themselves. What happens if I want to lookup bytestring-named attributes on a class defined in a Unicode-string module or vice versa?
        • im3w1l 5y ago
          I'm afraid I don't understand your question completely, so I'll answer what I guess you mean. Let's say you have a function in one module that expects a bytestring. This function is called from another module, and a unicode string is passed. What happens? Well python is a duck-typed language, so it will handle the unicode string as if it was a bytestring, and this may or may not work correctly. It's up to the modern code to interface correctly with the legacy code. This could mean manually converting between bytestring and unicode string.
    • ByteJockey 5y ago
      That ignores how python scoping works. Built-ins are actually the last thing python checks. You can override the built-ins by re-defining the symbol in any of: local functions, enclosing functions, or the global scope and it will impact the entirety of that scope.
      • pwdisswordfish8 5y ago
        You missed the point. It’s not about names themselves. What happens if I want to lookup bytestring-named attributes on a class defined in a Unicode-string module or vice versa?
        • ByteJockey 5y ago
          Ahh, the wording screwed me up. If I'm understanding your comment correctly this time, it would look something like: getattr(thing, b'attribute_name')
          • im3w1l 5y ago
            I guess you could make it work as long as everything is ascii right? And if it's not then raise exception.
            • ByteJockey 5y ago
              If it's not ascii, you would just do: getattr(thing, u'attribute_with_unicode_characters') This would work in both python 2.7 and 3.9 (probably works with other 3.X versions but it's the one I actually tried).
• chrismorgan 5y ago
  The changes that were needed were far too invasive for that to work in practice. Any library that touched anything even vaguely stringy or bytey would have needed to be operating in either bad mode (2) or better mode (3), but that would have been an almost-invisible part of its public API, which starts causing serious trouble when you want to use bad mode libraries and better mode libraries simultaneously, since the two are fairly incompatible, so that automatic translation would not be possible, so you’d have to know which mode a library was operating in, which undermines the whole endeavour.
  • dtech 5y ago
    In practice this isn't as big a problem as you're making it out to be, because UTF8 is compatible with ANSI. Nearly all of computing has made a similar transition in the 2000's and 2010's, and I've certainly debugged my share of encoding issues, but in general it worked for a large majority of use cases. I think most people would've taken a few encoding issues over the 10+ years long migration.
    • chrismorgan 5y ago
      If it were ASCII and UTF-8 that this was all about, it wouldn’t be as big a deal. But it’s not. Python 2 was a code pages world. Not an ASCII world. Also Python’s Unicode isn’t and hasn’t ever been UTF-8 anywhere; rather, it’s sequences of Unicode code points (note that that’s code points, not Unicode scalar values−this and the representation of str are a pair of things I find quite bafflingly bad about Python 3, because it means Python’s Unicode strings aren’t, y’know, Unicode strings, because they can be ill-formed, and they’re stuck with bad fixed-width representations so that any even vaguely interesting string will be stored as (ill-formed) UTF-32). Python 2’s str type was a mess, used for bytes data, for strings of uncertain encoding, and for Unicode strings. In a vacuum, you might say you could make that work by splitting it up (and indeed the types module was an early endeavour to do that sort of thing back before it became evident it was insufficient). The key is that this confusion and widespread misuse was rife through the standard library and other third-party libraries. If the Python 2 → 3 transition had been just str-as-bytes/str-as-who-knows-what/unicode → bytes/something/str, it might have been manageable. But the libraries were so shot that a straightforward migration was impossible, because str was too many things and it was quite impossible to tell statically what should be done about it. (And impossible dynamically to resolve as well.)