6 ms·

Except py3 needed to be a breaking change, because unicode handling. It doesn't matter if the API rearranging was delayed and the only change 2->3 was string/by

by kortex 5y ago

Except py3 needed to be a breaking change, because unicode handling. It doesn't matter if the API rearranging was delayed and the only change 2->3 was string/bytes handling, it still would have taken a decade. What's the alternative? Deprecate warning old-style string handling and slowly nudge it out? Great, that's not actually decideable, since you need to know how a str is being used. str are at the very heart of python. This would be a gargantuan task just to write the linter.

rst 5y ago
Ruby made a similar change in character encoding starting at roughly the same time; it's (barely) remembered now as a non-event because they didn't take the "opportunity" to bundle in a bunch of other breaking changes which collectively required much more pervasive dinks to pre-existing code than the encoding issues.
- masklinn 5y ago
  This and that have literally nothing to do with one another. AFAIK Ruby's change was that they introduced a notion of encoding to strings. Python literally change the string types and its semantics. Adding a bunch of other BC breakages was a good idea. Having been responsible for the migration of a large codebase the strings change was by far the most disruptive and problematic one, the rest did not even come close (the distant second was the change to rounding IIRC). Ripping the bandaid and adding a bunch of other BC breaks to clean things up or prepare for the future was I think an excellent idea. Most of those were easy to shim or work around (the syntactic BC breaks were especially trivial). The problem, as GGP's comments correctly notes, was the incompatibility between Python 2 and Python 3: when Python 3 was first released it was impossible to write a non-trivial program which could run on both version. Meaning if you wanted to port a project you had to convert everything at once. And as if the odds of that working were not infinitesimal enough, libraries had to fork their own codebases, and either try to maintain completely incompatible codebases concurrently or drop support for Python 2 while literally all their users were still on it.
- dragonwriter 5y ago
  > Ruby made a similar change in character encoding starting at roughly the same time; it's (barely) remembered now as a non-event because they didn't take the "opportunity" to bundle in a bunch of other breaking changes which collectively required much more pervasive dinks to pre-existing code than the encoding issues. No, its often misremembered that way; Ruby 1.9 had lots of backward-compatibility breaking changes, there was a long time when lots of things were stuck on 1.8, and much of the other stuff that happened with the Py2 -> Py3 conversion was mirrored. The big difference was that the Ruby community and ecosystem were smaller and less diverse.
  - kortex 5y ago
    Ruby's also a bad comparison because not many folks are running ruby on windows. Huge chunks of industry and academia run python on windows (usually through (ana/mini)conda). That's a whole 'nother world of encoding hurt.
    - dragonwriter 5y ago
      > Ruby's also a bad comparison because not many folks are running ruby on windows. Especially around the 1.9 transition; Ruby was barely usable beyond the level of toy code on windows. The Ruby on Windows story is actually a lot better now.
  - rst 5y ago
    Which things in particular? Rails supported both Ruby 1.8 and 1.9 simultaneously from Rails 2.3 through Rails 3.2, without it apparently being a huge struggle for the developers, dropping 1.8 only at about the same time that support for it was being dropped by the core Ruby devs altogether. And other posts from maintainers of smaller, but widely used, gems describe it as being a bit of a bother to set up, but not nearly the struggle that it took to run anything in both Python 2 and Python 3. See, e.g., https://thoughtbot.com/blog/the-journey-to-ruby-1-9 https://thoughtbot.com/blog/the-journey-to-ruby-1-9 One key here is that the common subset of Ruby 1.8 and 1.9 was large enough that you could ship a single library for both, and a lot of large ones did, as in the above examples. That wasn't the case initially in the Python transition, and by the time Python3 had started making enough concessions to back compatibility to make this practically possible, people were a lot less willing to try. (I personally was managing upgrades or a large Rails app at the time; Rails upgrades were, and remain, a bear, but as an accident of code style which just avoided the tricky spots anyway, the changes required to move that code between any two Ruby versions have never been more than a day or two.)
TheCoelacanth 5y ago
It didn't need to be such a disruptive breaking change that fragmented the ecosystem. They should have made the unicode handling changes opt-in per package or file. Then, start warning about anything that hasn't opted in. Then, only after a significant migration period, change the default to the new unicode handling.
- im3w1l 5y ago
  It's well known that you should make the "correct" choice the default. So unicode should have been opt-out (from __past__ import legacy_strings). Result: Everyone batch-adds unicode opt-out to their whole codebase, but new files use the modern way. Gradually people remove the opt outs as they go over old code.
  - TheCoelacanth 5y ago
    The correct way should be the default, but in the case where you already chose the wrong default, you shouldn't introduce the correct way of handling things and make it the default all at once. Upgrading to a new major version should never require changes to your code if you fixed all deprecation warnings in the previous major version. Otherwise, you make incremental upgrades impossible. The new alternative should always be introduced as opt-in with the old default deprecated, and then made default in the next major version.
  - pwdisswordfish8 5y ago
    This couldn’t work. For starters, the names of classes, attributes and functions are str in both versions, which is to say, bytestrings in Python 2 and Unicode strings in Python 3. This cannot be handwaved away with a pragma.
    - im3w1l 5y ago
      str = int is perfectly valid python. So having str mean different things in different modules is already possible.
marcosdumay 5y ago
> What's the alternative? Have an import switch that enables a legacy mode, that will convert your strings into and out of the library? You can even require that the application developers cast their strings into the correct ones if you don't want to calculate it on the compiler.
- kortex 5y ago
  That's just not possible with python's computational model. That works for "from future import print" cause that's just an additional function. str is too deeply embedded and used in too many dynamic and unique ways for that approach to work without doing some really ugly or nonperformant or community-fracturing things.
masklinn 5y ago
You're completely missing the point of GP's comment. Their point is not that Python3 should not have broken backwards compatibility, it's that Python 3 should not have been entirely incompatible with Python 2. That decision is what hampered migration for years, because it turned out not to be a feasible migration strategy, it would have required the entire ecosystem to migrate in lockstep which when you think about it made no sense. Python 3 migration picked up steam once the core team backported what could be backported to Python 2(.7), and reintroduced syntactic compatibility features to Python 3. At that point, and with a few shims for standard library stuff, libraries could create cross-version codebases, which meant they didn't have to try and fork their own codebases. And dependents could migrate at their leisure.
- kortex 5y ago
  > Their point is not that Python3 should not have broken backwards compatibility, it's that Python 3 should not have been entirely incompatible with Python 2. No, my point is precisely that py2 and 3 are not compatible. Even with the backports, py2 and 3 are not fully compatible. Hence the "py23" or "six" dialect, which is the subset of 2 and 3 that overlap, with some shims to facilitate it. Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work. Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model. Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route?
  - masklinn 5y ago
    > No, my point is precisely that py2 and 3 are not compatible. Which still misses the point, py2 and py3 not being compatible is different than them being completely incompatible, because there is a useful, workable, common subset of Python 2 and Python 3. The entire issue with Python 3 was that there originally was not, the core team had to be dragged kicking and screaming into carving it out through backports and reintroductions because their original strategy was not workable. > Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work. So? > Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model. How can you claim that when "shimming their way between 2 and 3" is exactly what ended up happening? > Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route? Have you considered stopping beating up your strawman?

by kortex 5y ago

rst 5y ago

Ruby made a similar change in character encoding starting at roughly the same time; it's (barely) remembered now as a non-event because they didn't take the "opportunity" to bundle in a bunch of other breaking changes which collectively required much more pervasive dinks to pre-existing code than the encoding issues.

masklinn 5y ago
This and that have literally nothing to do with one another. AFAIK Ruby's change was that they introduced a notion of encoding to strings. Python literally change the string types and its semantics. Adding a bunch of other BC breakages was a good idea. Having been responsible for the migration of a large codebase the strings change was by far the most disruptive and problematic one, the rest did not even come close (the distant second was the change to rounding IIRC). Ripping the bandaid and adding a bunch of other BC breaks to clean things up or prepare for the future was I think an excellent idea. Most of those were easy to shim or work around (the syntactic BC breaks were especially trivial). The problem, as GGP's comments correctly notes, was the incompatibility between Python 2 and Python 3: when Python 3 was first released it was impossible to write a non-trivial program which could run on both version. Meaning if you wanted to port a project you had to convert everything at once. And as if the odds of that working were not infinitesimal enough, libraries had to fork their own codebases, and either try to maintain completely incompatible codebases concurrently or drop support for Python 2 while literally all their users were still on it.
dragonwriter 5y ago
> Ruby made a similar change in character encoding starting at roughly the same time; it's (barely) remembered now as a non-event because they didn't take the "opportunity" to bundle in a bunch of other breaking changes which collectively required much more pervasive dinks to pre-existing code than the encoding issues. No, its often misremembered that way; Ruby 1.9 had lots of backward-compatibility breaking changes, there was a long time when lots of things were stuck on 1.8, and much of the other stuff that happened with the Py2 -> Py3 conversion was mirrored. The big difference was that the Ruby community and ecosystem were smaller and less diverse.
- kortex 5y ago
  Ruby's also a bad comparison because not many folks are running ruby on windows. Huge chunks of industry and academia run python on windows (usually through (ana/mini)conda). That's a whole 'nother world of encoding hurt.
  - dragonwriter 5y ago
    > Ruby's also a bad comparison because not many folks are running ruby on windows. Especially around the 1.9 transition; Ruby was barely usable beyond the level of toy code on windows. The Ruby on Windows story is actually a lot better now.
- rst 5y ago
  Which things in particular? Rails supported both Ruby 1.8 and 1.9 simultaneously from Rails 2.3 through Rails 3.2, without it apparently being a huge struggle for the developers, dropping 1.8 only at about the same time that support for it was being dropped by the core Ruby devs altogether. And other posts from maintainers of smaller, but widely used, gems describe it as being a bit of a bother to set up, but not nearly the struggle that it took to run anything in both Python 2 and Python 3. See, e.g., https://thoughtbot.com/blog/the-journey-to-ruby-1-9 https://thoughtbot.com/blog/the-journey-to-ruby-1-9 One key here is that the common subset of Ruby 1.8 and 1.9 was large enough that you could ship a single library for both, and a lot of large ones did, as in the above examples. That wasn't the case initially in the Python transition, and by the time Python3 had started making enough concessions to back compatibility to make this practically possible, people were a lot less willing to try. (I personally was managing upgrades or a large Rails app at the time; Rails upgrades were, and remain, a bear, but as an accident of code style which just avoided the tricky spots anyway, the changes required to move that code between any two Ruby versions have never been more than a day or two.)

TheCoelacanth 5y ago

It didn't need to be such a disruptive breaking change that fragmented the ecosystem. They should have made the unicode handling changes opt-in per package or file. Then, start warning about anything that hasn't opted in. Then, only after a significant migration period, change the default to the new unicode handling.

im3w1l 5y ago
It's well known that you should make the "correct" choice the default. So unicode should have been opt-out (from __past__ import legacy_strings). Result: Everyone batch-adds unicode opt-out to their whole codebase, but new files use the modern way. Gradually people remove the opt outs as they go over old code.
- TheCoelacanth 5y ago
  The correct way should be the default, but in the case where you already chose the wrong default, you shouldn't introduce the correct way of handling things and make it the default all at once. Upgrading to a new major version should never require changes to your code if you fixed all deprecation warnings in the previous major version. Otherwise, you make incremental upgrades impossible. The new alternative should always be introduced as opt-in with the old default deprecated, and then made default in the next major version.
- pwdisswordfish8 5y ago
  This couldn’t work. For starters, the names of classes, attributes and functions are str in both versions, which is to say, bytestrings in Python 2 and Unicode strings in Python 3. This cannot be handwaved away with a pragma.
  - im3w1l 5y ago
    str = int is perfectly valid python. So having str mean different things in different modules is already possible.

marcosdumay 5y ago

> What's the alternative? Have an import switch that enables a legacy mode, that will convert your strings into and out of the library? You can even require that the application developers cast their strings into the correct ones if you don't want to calculate it on the compiler.

kortex 5y ago
That's just not possible with python's computational model. That works for "from future import print" cause that's just an additional function. str is too deeply embedded and used in too many dynamic and unique ways for that approach to work without doing some really ugly or nonperformant or community-fracturing things.

masklinn 5y ago

You're completely missing the point of GP's comment. Their point is not that Python3 should not have broken backwards compatibility, it's that Python 3 should not have been entirely incompatible with Python 2. That decision is what hampered migration for years, because it turned out not to be a feasible migration strategy, it would have required the entire ecosystem to migrate in lockstep which when you think about it made no sense. Python 3 migration picked up steam once the core team backported what could be backported to Python 2(.7), and reintroduced syntactic compatibility features to Python 3. At that point, and with a few shims for standard library stuff, libraries could create cross-version codebases, which meant they didn't have to try and fork their own codebases. And dependents could migrate at their leisure.

kortex 5y ago
> Their point is not that Python3 should not have broken backwards compatibility, it's that Python 3 should not have been entirely incompatible with Python 2. No, my point is precisely that py2 and 3 are not compatible. Even with the backports, py2 and 3 are not fully compatible. Hence the "py23" or "six" dialect, which is the subset of 2 and 3 that overlap, with some shims to facilitate it. Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work. Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model. Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route?
- masklinn 5y ago
  > No, my point is precisely that py2 and 3 are not compatible. Which still misses the point, py2 and py3 not being compatible is different than them being completely incompatible, because there is a useful, workable, common subset of Python 2 and Python 3. The entire issue with Python 3 was that there originally was not, the core team had to be dragged kicking and screaming into carving it out through backports and reintroductions because their original strategy was not workable. > Py2 and Py3 string apis are not compatible. They cannot be made compatible. They can never be compatible. There's no "from __future__" that could make them compatible without a stupendous amount of work. So? > Not to be pretentious, but I'm developing the opinion that anyone who thinks that the Python team could have just shimmed their way between 2 and 3, does not really understand the Python compute model. How can you claim that when "shimming their way between 2 and 3" is exactly what ended up happening? > Google literally invented a py2 to go transpiler (grumpy) to avoid converting. Do you think for a second if there was a way to just "from future" their way to a compatible str object, they wouldn't have taken that route? Have you considered stopping beating up your strawman?