5 ms·

As mentioned by several people already, the migration process from 2 to 3 was really painful. It will be easier for the Python core maintainers, but introducin

As mentioned by several people already, the migration process from 2 to 3 was really painful. It will be easier for the Python core maintainers, but introducing a breaking change with the intent that that version 2 will be deprecated has probably caused more distress overall, especially maintainers of other open source projects which rely on a specific python version. Let's never have the same kind of breaking change again, please.

lifthrasiir 4y ago
> It will be easier for the Python core maintainers, [...] It wasn't not even that easy, to be frank. 2to3 is not a small project [1] and they had (alas, incorrectly) assumed that there were enough people to have enough tests to be benefited from 2to3, only to be found massively wrong. In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it. [1] https://github.com/python/cpython/tree/main/Lib/lib2to3 https://github.com/python/cpython/tree/main/Lib/lib2to3
- kasabali 4y ago
  > first major language with this approach was ECMAScript 2009 (5th ed.) Not that I've any proof but I kind of find it hard to believe there were no other examples before
  - TJSomething 4y ago
    Perl's original "use strict" was added in 1994 with Perl 5, to allow backward compatibility with Perl 4 scripts.
  - masklinn 4y ago
    Python had that long before js, with __future__ imports. But editions & friends are for syntax. Python 3 changed semantics all around. Editions are not an option when you’re changing the langage’s entire text model, especially without static typing.
    - jsmeaton 4y ago
      Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it. Python 3.1 could then have introduced some of the other minor changes, perhaps behind a future import. I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. We couldn’t have done it without six. It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
      - masklinn 4y ago
        > Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it. Except "that's it" does a ton of work here, as the entire stdlib had to be audited and updated to correctly work with the new text model, plus other semantics change (e.g. the removal of total ordering). That those were so large and wide-ranging in the first place is why the core team decided that fixing a bunch of other issues could go in as well e.g. reorganise the stdlib, move keywords to functions, remove deprecated stuff, etc... Those ancillary changes were mostly pretty easy to implement. > I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. It wouldn't have been more pleasant if the breaking changes had been spread over multiple versions, you'd still have eaten all of those, and the early adopter would have gotten even more pain as they'd have needed to implement breaking changes on every update. > We couldn’t have done it without six. We did just fine without six migrating a hundreds-kloc-scale repo from 2.7 to 3.5, we basically copied what Werkzeug did, with our own "minisix" module for just the bits we needed (some lifted from werkzeug's and others bespoke e.g. we needed a compatibility layer for CSV). We could not, however, have done without the 3.0 - 3.3 improvements, the existence of 2.7, or the community experience. And tests. Having a well-tested codebase was absolutely essential to a comfy migration, the less a corner of the codebase was tested, the more migration bug kept popping up afterwards. > It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake. No, doing all those breaking changes version after version would have been a lot worse. IME the vast, vast majority of the pain (short and long term) came from the semantics changes, not the syntactic ones. Those were relatively easy: do the syntactic change, enable the corresponding lint (so colleagues don't reintroduce the old construct out of habit), commit. A few of the API changes were frustrating (like base64 which was clearly mis-converted by the core team but what do you do). - the text model stuff took by far the longest and had by far the longest tail (as in things we missed during the migration), this is in part because not all of the changes or Python 3 APIs are ideal, but mostly it's just a lot of runtime things in a lot of paths if your software leverages strings a lot - total ordering removal broke a surprising amount of things, and also had a pretty long tail of bugs, mostly in that `None` would get mixed in with other values and get fed to comparison operators, sorting, or min/max - `round` was a surprise contender for pain in the ass, not just because it changed the rounding method (to banker's rounding, from I don't remember what) but also because P2's round() always returned floats while P3 returns integers if precision is missing / None - changes to dict iteration also broke a few things, there were multiple pieces of code with implicit reliance on iteration order, although for the most part they were issues in numerical stability of floats, which broke tests using strict equality on floats - division changes also broke a few things as it wasn't always easy to recognise whether operands were always floats or could be int, also `//` being floor division rather than integer division led to a few behavioural changes We also did a few things like banning the builtins which had changed behaviour (open, map, filter, ...) for the duration of the migration, the issues they caused were subtle enough as to not be worth it. They were re-allowed once we dropped P2.
- kazinator 4y ago
  The idea of compilers and interpreters providing backward compatibility options to support programs written using older dialects did not originate in Javascript.
- masklinn 4y ago
  > (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it. https://docs.python.org/3/library/__future__.html https://docs.python.org/3/library/__future__.html That only works with “syntactic” updates, not with deep semantics changes. Those are why the core team pulled the trigger on an overhaul update. And they were by far the biggest issues during migration, anything which would be gated behind an edition or opt-in was easy.
- ilyt 4y ago
  > In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it. Perl did it since ages. You just put use v5.24 in header and interpreter enabled those language features on file by file basis
  - Conan_Kudo 4y ago
    Perl's behavior was utterly broken. It worked more like how a C/C++ compiler handles language standards (by lying to you) than actually enforcing behaviors. From the toolchain and language developer side, nobody liked Perl's approach. That's why there was so much effort around Perl 6 (which became Raku) at around the same time.
    - sigzero 4y ago
      That isn't even remotely true and definitely was not why so much effort into Perl 6 happened. Perl 6 was a different language with Perl-ish semantics and why it was renamed to Raku to get it out of the Perl namespace. The "use version" behavior has been used forever and is not "utterly broken". It isn't even "broken" at all.
  - toast0 4y ago
    At Yahoo, we had a wrapper so we could have 5.6 and 5.8 installed (and later 5.8 and 5.10), and if you had use 5.6, you'd get 5.6, and use 5.8, you'd get 5.8, and neither, you'd get the default set for that machine. If you tried to run a 5.8 script on a machine without 5.8, you'd get an error as should be expected. If you ran a use 5.6 script on a 5.8/5.10 machine, I think you'd get 5.8, but I can't remember... 5.10 was less disruptive than 5.8.
- deleted 4y ago
  [deleted]
yen223 4y ago
I see people saying that the migration was painful, but my own experience with two separate Python 2->3 migrations were so smooth they might as well have been a minor version bumps. I don't know where this discrepancy comes from.
- TylerE 4y ago
  How big were these projects? I work on a giant py2 codebqse with thousands of _files_. I haven’t run sloccount in a while but I bet we’re over 1M LOC. (And before anyone asks, a big part of why it’s so big is that it’s very old. So old it predates Django, so, in house framework, in house ORM, etc.)
- vkazanov 4y ago
  Some time ago we migrated a few hundred thousands lines of Py2 spread over dozens of services. Easiest case possible: every service can be converted and tested separately. Took a calendar year of ca-a-a-arefully migrating these one by one, all while delivering stuff and making sure nothing falls apart along the way. Not pleasant. Can't recommend.
- xdennis 4y ago
  It was easy if you were working on your own packages. It was quite difficult if you were dependent on external packages that had no plan to upgrade or if your company had internal packages which were Python 2-only and required in most services.
- Joeboy 4y ago
  I migrated two significant projects, one of them took about half a day on my own, the other took several months with a team of people contributing.
  - deleted 4y ago
    [deleted]
- xnorswap 4y ago
  The migration as a whole has been painful. It's been 14 years and dropping python2 is still controversial. It's a story of how not to do a major language upgrade. It's not quite as bad as "Perl 6" which killed off the whole language, but it's up there with massive mistakes which has cost the industry dearly.
pixelfarmer 4y ago
Python 3.0 was released almost 15 years ago and it was clear at that point that Python 2 would be shown the door eventually. My empathy for any complaints regarding that switch is exactly 0, even for open source projects.
- whirlwin 4y ago
  There are many core Python projects that predates Python 3. We cannot deny everything that happend before Python 3 was released.
  - pixelfarmer 4y ago
    I've had to deal with a lot of stupidity regarding Python, and that started with assuming that /usr/bin/python is Python 2. Why? Because that was the default in distros for just way too long, and some of the problems only got fixed after the distros went with Python 3 as the default. Obviously some people need to get the rug pulled under their feet before they start moving.
willis936 4y ago
Breaking changes has been and will continue to be the pythonic way. I avoid it.

lifthrasiir 4y ago

> It will be easier for the Python core maintainers, [...] It wasn't not even that easy, to be frank. 2to3 is not a small project [1] and they had (alas, incorrectly) assumed that there were enough people to have enough tests to be benefited from 2to3, only to be found massively wrong. In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it. [1] https://github.com/python/cpython/tree/main/Lib/lib2to3 https://github.com/python/cpython/tree/main/Lib/lib2to3

kasabali 4y ago
> first major language with this approach was ECMAScript 2009 (5th ed.) Not that I've any proof but I kind of find it hard to believe there were no other examples before
- TJSomething 4y ago
  Perl's original "use strict" was added in 1994 with Perl 5, to allow backward compatibility with Perl 4 scripts.
- masklinn 4y ago
  Python had that long before js, with __future__ imports. But editions & friends are for syntax. Python 3 changed semantics all around. Editions are not an option when you’re changing the langage’s entire text model, especially without static typing.
  - jsmeaton 4y ago
    Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it. Python 3.1 could then have introduced some of the other minor changes, perhaps behind a future import. I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. We couldn’t have done it without six. It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake.
    - masklinn 4y ago
      > Could they have at least minimised the blast radius? For example, make all of the text/Unicode changes for Python 3 and thats it. Except "that's it" does a ton of work here, as the entire stdlib had to be audited and updated to correctly work with the new text model, plus other semantics change (e.g. the removal of total ordering). That those were so large and wide-ranging in the first place is why the core team decided that fixing a bunch of other issues could go in as well e.g. reorganise the stdlib, move keywords to functions, remove deprecated stuff, etc... Those ancillary changes were mostly pretty easy to implement. > I had a hand in moving from 2.7 to 3.6 for a 1M LOC repo and it was not pleasant. So many backward compatible changes that that all had to be accounted for rather than being able to focus on the major breaking change of a particular version. It wouldn't have been more pleasant if the breaking changes had been spread over multiple versions, you'd still have eaten all of those, and the early adopter would have gotten even more pain as they'd have needed to implement breaking changes on every update. > We couldn’t have done it without six. We did just fine without six migrating a hundreds-kloc-scale repo from 2.7 to 3.5, we basically copied what Werkzeug did, with our own "minisix" module for just the bits we needed (some lifted from werkzeug's and others bespoke e.g. we needed a compatibility layer for CSV). We could not, however, have done without the 3.0 - 3.3 improvements, the existence of 2.7, or the community experience. And tests. Having a well-tested codebase was absolutely essential to a comfy migration, the less a corner of the codebase was tested, the more migration bug kept popping up afterwards. > It seems that a bunch of changes were made all at once because it was already going to be incompatible so let’s take the opportunity to clean up as much as possible. I think that was the big mistake. No, doing all those breaking changes version after version would have been a lot worse. IME the vast, vast majority of the pain (short and long term) came from the semantics changes, not the syntactic ones. Those were relatively easy: do the syntactic change, enable the corresponding lint (so colleagues don't reintroduce the old construct out of habit), commit. A few of the API changes were frustrating (like base64 which was clearly mis-converted by the core team but what do you do). - the text model stuff took by far the longest and had by far the longest tail (as in things we missed during the migration), this is in part because not all of the changes or Python 3 APIs are ideal, but mostly it's just a lot of runtime things in a lot of paths if your software leverages strings a lot - total ordering removal broke a surprising amount of things, and also had a pretty long tail of bugs, mostly in that `None` would get mixed in with other values and get fed to comparison operators, sorting, or min/max - `round` was a surprise contender for pain in the ass, not just because it changed the rounding method (to banker's rounding, from I don't remember what) but also because P2's round() always returned floats while P3 returns integers if precision is missing / None - changes to dict iteration also broke a few things, there were multiple pieces of code with implicit reliance on iteration order, although for the most part they were issues in numerical stability of floats, which broke tests using strict equality on floats - division changes also broke a few things as it wasn't always easy to recognise whether operands were always floats or could be int, also `//` being floor division rather than integer division led to a few behavioural changes We also did a few things like banning the builtins which had changed behaviour (open, map, filter, ...) for the duration of the migration, the issues they caused were subtle enough as to not be worth it. They were re-allowed once we dropped P2.
kazinator 4y ago
The idea of compilers and interpreters providing backward compatibility options to support programs written using older dialects did not originate in Javascript.
masklinn 4y ago
> (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it. https://docs.python.org/3/library/__future__.html https://docs.python.org/3/library/__future__.html That only works with “syntactic” updates, not with deep semantics changes. Those are why the core team pulled the trigger on an overhaul update. And they were by far the biggest issues during migration, anything which would be gated behind an edition or opt-in was easy.
ilyt 4y ago
> In hindsight no one had a good idea about language evolution; the modern concensus is a single interpreter supporting multiple parallel versions, also called "editions", but (EDIT: to my knowledge) the first major language with this approach was ECMAScript 2009 (5th ed.) with `use strict` and Python 3 predated it. Perl did it since ages. You just put use v5.24 in header and interpreter enabled those language features on file by file basis
- Conan_Kudo 4y ago
  Perl's behavior was utterly broken. It worked more like how a C/C++ compiler handles language standards (by lying to you) than actually enforcing behaviors. From the toolchain and language developer side, nobody liked Perl's approach. That's why there was so much effort around Perl 6 (which became Raku) at around the same time.
  - sigzero 4y ago
    That isn't even remotely true and definitely was not why so much effort into Perl 6 happened. Perl 6 was a different language with Perl-ish semantics and why it was renamed to Raku to get it out of the Perl namespace. The "use version" behavior has been used forever and is not "utterly broken". It isn't even "broken" at all.
- toast0 4y ago
  At Yahoo, we had a wrapper so we could have 5.6 and 5.8 installed (and later 5.8 and 5.10), and if you had use 5.6, you'd get 5.6, and use 5.8, you'd get 5.8, and neither, you'd get the default set for that machine. If you tried to run a 5.8 script on a machine without 5.8, you'd get an error as should be expected. If you ran a use 5.6 script on a 5.8/5.10 machine, I think you'd get 5.8, but I can't remember... 5.10 was less disruptive than 5.8.
deleted 4y ago
[deleted]

yen223 4y ago

I see people saying that the migration was painful, but my own experience with two separate Python 2->3 migrations were so smooth they might as well have been a minor version bumps. I don't know where this discrepancy comes from.

TylerE 4y ago
How big were these projects? I work on a giant py2 codebqse with thousands of _files_. I haven’t run sloccount in a while but I bet we’re over 1M LOC. (And before anyone asks, a big part of why it’s so big is that it’s very old. So old it predates Django, so, in house framework, in house ORM, etc.)
vkazanov 4y ago
Some time ago we migrated a few hundred thousands lines of Py2 spread over dozens of services. Easiest case possible: every service can be converted and tested separately. Took a calendar year of ca-a-a-arefully migrating these one by one, all while delivering stuff and making sure nothing falls apart along the way. Not pleasant. Can't recommend.
xdennis 4y ago
It was easy if you were working on your own packages. It was quite difficult if you were dependent on external packages that had no plan to upgrade or if your company had internal packages which were Python 2-only and required in most services.
Joeboy 4y ago
I migrated two significant projects, one of them took about half a day on my own, the other took several months with a team of people contributing.
- deleted 4y ago
  [deleted]
xnorswap 4y ago
The migration as a whole has been painful. It's been 14 years and dropping python2 is still controversial. It's a story of how not to do a major language upgrade. It's not quite as bad as "Perl 6" which killed off the whole language, but it's up there with massive mistakes which has cost the industry dearly.

pixelfarmer 4y ago

Python 3.0 was released almost 15 years ago and it was clear at that point that Python 2 would be shown the door eventually. My empathy for any complaints regarding that switch is exactly 0, even for open source projects.

whirlwin 4y ago
There are many core Python projects that predates Python 3. We cannot deny everything that happend before Python 3 was released.
- pixelfarmer 4y ago
  I've had to deal with a lot of stupidity regarding Python, and that started with assuming that /usr/bin/python is Python 2. Why? Because that was the default in distros for just way too long, and some of the problems only got fixed after the distros went with Python 3 as the default. Obviously some people need to get the rug pulled under their feet before they start moving.

willis936 4y ago

Breaking changes has been and will continue to be the pythonic way. I avoid it.