6 ms·

I really feel like the community was a difficult pain in the ass during the 2->3 migration. There were breaking changes but I’m sure the Python maintainers didn

by thiht 4y ago

I really feel like the community was a difficult pain in the ass during the 2->3 migration. There were breaking changes but I’m sure the Python maintainers didn’t expect the community to react so badly. Are there some valid reasons that made 2->3 migration insanely hard for some projects? I remember seeing blog articles whining about print vs print(), but surely there are some more important stuff.

pjc50 4y ago
> Are there some valid reasons that made 2->3 migration insanely hard for some projects? Print was a non-issue once you could do "from __future__ import print_function". It was the unicode migration. All string handling from I/O required at least checking. The dependency issue was the most crippling, though: because you can't load Python 2 code from Python 3, before your project can begin migrating you have to wait for all your dependencies to update as well. This may involve them changing the types that get passed over the API from 'str' to 'bytes' or vice versa, which is potentially a breaking change.
- nonethewiser 4y ago
  Sounds like a circular problem
MereInterest 4y ago
Migrating where there’s a 1-1 mapping of old API to new API is easy, such as the print to print() change. The hard part is when there’s a 1-N mapping, where a single name in the old API becomes multiple names in the new API. These are the most useful changes to make, because they can split apart unrelated concepts that had been erroneously represented by the same object, but require a human to determine which use case applies. For python2, the str class could either contain a sequence of characters, or could contain a sequence of bytes. These are often conflated when using ASCII encoded text, but are very different for any other encoding. In python3, these are represented by the str class for a sequence of characters, and by the bytes class for a sequence of bytes. It’s so much nicer to work with, but required manually separating out the earlier use case.
- otabdeveloper4 4y ago
  Python2 had the 'unicode' class for characters, and the 'str' class for bytes. The whole 2 to 3 debacle was only because somebody thought that naming your string class 'unicode' doesn't sound nice enough or something.
  - Izkata 4y ago
    Also the default, what a no-prefix string like "foo" meant changed between the two versions.
    - someguydave 4y ago
      Right, essentially the Python developers decided to define a sizable portion of their developer base as "bad people" due to their particular use of the string API and then spent years going to war with them. The war continues to this day apparently with this Debian news. Somehow you can use fortran77 compilers without people getting angry but you can't use Python2
  - acdha 4y ago
    It’s not that simple: as the person you’re responding to explained, this was leaky because in simple cases you could pretend those were equivalent but then things would break around interface points when someone introduced things like filenames with Unicode. I had clean Python 2 codebases which had been Unicode-safe for years and had good test coverage, so porting to Python 3 was trivial and mostly automated. The projects which were hard were the sloppy ones which conflated bytes and Unicode, and the poor development culture which lead to that also tended to mean they had limited testing, orphaned dependencies, monkey patching, etc. which made the migration hard. I also found a fair number of cases where that process fixed other bugs which had been ignored for years.
qbasic_forever 4y ago
Guido has done some pycon keynotes in the recent past and said himself he takes some of the blame for the transition. He says he greatly underestimated the impact of breaking changes and the inertia to get the community to change code. In particular not shipping tools to help automate the migration, or even thinking through a migration path with targeted backwards compatibility/back ports from the start (these eventually came quite a few releases and years into the transition). I don't think it's fair to say the community was entirely at fault for the length of time. edit: This talk he did from pycascades 2018 in particular I remember seeing and being a good retrospective: https://www.youtube.com/watch?v=Oiw23yfqQy8 https://www.youtube.com/watch?v=Oiw23yfqQy8
- k__ 4y ago
  Codemods aren't a thing in Python?
  - qbasic_forever 4y ago
    They are a thing, the ast module was created to help with the transition and tooling to automate some of the conversion: https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html IIRC it wasn't there at the start though.
  - marssaxman 4y ago
    What is a "codemod"?
    - deleted 4y ago
      [deleted]
    - charcircuit 4y ago
      A codemod is an automated refactor. Codemods are typically done by writing scripts that manipulate the AST of code in files or by writing a regex to find and replace. The name originates from an internal script at Facebook called codemod. It had a public open source fork if you are curious. https://github.com/facebookarchive/codemod https://github.com/facebookarchive/codemod
onphonenow 4y ago
The python maintainers went out of their way to break things - and were smug about it. That’s what it felt like to the community. A big part of 3 was Unicode strings. In 2 you could mark a string as Unicode with a u””. This would have been a great way to let libraries and code work with both 2 and 3. They banned this, but you had to still use b””. This was the attitude and just one example. The list of stuff that got hard / slow was long. Migration path was unnecessarily difficult as was compatible code. Somewhere around 3.4 it was like a light switch flipped and they started being more reasonable. ASCII handling improved (yes, not all internet protocol stuff is Unicode), they began making it easier to target 2/3 etc. but it was horrible to start. I’ve thankfully forgotten some of the details :)
- bjt2n3904 4y ago
  The era of "move fast and break things" was disastrous for a lot of things.
  - acdha 4y ago
    12 years is moving fast?
- simplotek 4y ago
  > Migration path was unnecessarily difficult as was compatible code. I feel this take is outright wrong. Python provided it's 2to3 tool[1] which took care of the bulk of the work required to port Python2 code to Python3. The only code that was not supported was eggregious errors in the code that worked by coincidence, such as handling bytes as strings and vice-versa. Porting old code to Python3 is a breeze that consisted of running 2to3, run tests with python3, and if anything breaked then just touch up the code to get it to work. I know it because I personally ported half a dozen projects throughout the years. I'll go as far as to claim that most problems porting old python code to 3 were either upstream dependencies dragging their feet or internal human/organizational issues. [1] https://docs.python.org/3/library/2to3.html https://docs.python.org/3/library/2to3.html
  - onphonenow 4y ago
amiga386 4y ago
The biggest problem is fundamental to Python and a lot of other scripting languages: no static typing. Running Python 2 code direct on Python 3? All looks good, until _somewhere_ you use a method of str that isn't on bytes, or vice versa, or something that was a list before is now an iterable and it blows up in your face... but _only_ if code execution reaches that line. So you have to test _everything_. And if you're a library/module, that means testing with _every user_. Secondly, it seems to me the Python team didn't want to make it possible to write code that works correctly on both Python 2 and 3 simultaneously. They wanted you to write code that only works on Python 3 and tell every single Python 2 user to get lost. Essentially, they wanted to co-opt you as their agent of change; burn your own reputation by abandoning your userbase (or worse, start proselytising Py3 at them) or double your own workload to maintain two codebases. Who wants that? It's fundamentally an attitude problem. A desire for there to be only one right way to do things, a change of mind of what the right way is, and kicking the chair out from anyone doing it the old right way; the only compromise being to kicking the chair very very slowly. In Java, you can still run code that uses Vector and Hashtable from 1996, they're not even deprecated. They're used by _very_ little code since generic List and Map APIs superceded them 2004, but they're still there.
xorcist 4y ago
The early Python 3 releases were abysmal, much slower than Python 2 even without considering the bugs. That alone was bad. But there was also real world problems, like how some syscall wrappers would not take bytestrings where they should have, so you literally could not open files unless you had a valid unicode representation of their file name. That's fun when all you want to do is some data processing. The core team was swamped with other work around the migration and the community found their response to real world problems lacking. They did things like declaring Python 2 dead long before 3 was usable. Then they refused to fix SSL problems with Python 2, which everyone desperately needed, and went out of their way to oppose a community-led release to fix those problems. It all ended well. There was another Python 2 release, and Python 3.4 came and fixed most problems people had. It was still slower, but performance picked up and by 3.7 it was just as good if not better. But it was a good ten years during which the response to real world issues could have been a lot better. All in all, the migration could have happened much smoother. They could have learned from other languages. But ten years is not unreasonable for backwards incompatible changes for a major programming language. The idea that the process could be accelerated by neglecting the old version does not work. In my personal opinion, the single thing that could have made the transition better for the community would have been the possibility to run Python 2 code in Python 3. It was discussed a lot before the transition and was deemed impossible because of conflicting data types. But it's not impossible, it's just (a lot of) work. It's another type conversion, and Python can already do those. The impossibility to link to old libraries made the conversion a flag day for most code bases, and those are really hard.

by thiht 4y ago

pjc50 4y ago

> Are there some valid reasons that made 2->3 migration insanely hard for some projects? Print was a non-issue once you could do "from __future__ import print_function". It was the unicode migration. All string handling from I/O required at least checking. The dependency issue was the most crippling, though: because you can't load Python 2 code from Python 3, before your project can begin migrating you have to wait for all your dependencies to update as well. This may involve them changing the types that get passed over the API from 'str' to 'bytes' or vice versa, which is potentially a breaking change.

nonethewiser 4y ago
Sounds like a circular problem

MereInterest 4y ago

Migrating where there’s a 1-1 mapping of old API to new API is easy, such as the print to print() change. The hard part is when there’s a 1-N mapping, where a single name in the old API becomes multiple names in the new API. These are the most useful changes to make, because they can split apart unrelated concepts that had been erroneously represented by the same object, but require a human to determine which use case applies. For python2, the str class could either contain a sequence of characters, or could contain a sequence of bytes. These are often conflated when using ASCII encoded text, but are very different for any other encoding. In python3, these are represented by the str class for a sequence of characters, and by the bytes class for a sequence of bytes. It’s so much nicer to work with, but required manually separating out the earlier use case.

otabdeveloper4 4y ago
Python2 had the 'unicode' class for characters, and the 'str' class for bytes. The whole 2 to 3 debacle was only because somebody thought that naming your string class 'unicode' doesn't sound nice enough or something.
- Izkata 4y ago
  Also the default, what a no-prefix string like "foo" meant changed between the two versions.
  - someguydave 4y ago
    Right, essentially the Python developers decided to define a sizable portion of their developer base as "bad people" due to their particular use of the string API and then spent years going to war with them. The war continues to this day apparently with this Debian news. Somehow you can use fortran77 compilers without people getting angry but you can't use Python2
- acdha 4y ago
  It’s not that simple: as the person you’re responding to explained, this was leaky because in simple cases you could pretend those were equivalent but then things would break around interface points when someone introduced things like filenames with Unicode. I had clean Python 2 codebases which had been Unicode-safe for years and had good test coverage, so porting to Python 3 was trivial and mostly automated. The projects which were hard were the sloppy ones which conflated bytes and Unicode, and the poor development culture which lead to that also tended to mean they had limited testing, orphaned dependencies, monkey patching, etc. which made the migration hard. I also found a fair number of cases where that process fixed other bugs which had been ignored for years.

qbasic_forever 4y ago

Guido has done some pycon keynotes in the recent past and said himself he takes some of the blame for the transition. He says he greatly underestimated the impact of breaking changes and the inertia to get the community to change code. In particular not shipping tools to help automate the migration, or even thinking through a migration path with targeted backwards compatibility/back ports from the start (these eventually came quite a few releases and years into the transition). I don't think it's fair to say the community was entirely at fault for the length of time. edit: This talk he did from pycascades 2018 in particular I remember seeing and being a good retrospective: https://www.youtube.com/watch?v=Oiw23yfqQy8 https://www.youtube.com/watch?v=Oiw23yfqQy8

k__ 4y ago
Codemods aren't a thing in Python?
- qbasic_forever 4y ago
  They are a thing, the ast module was created to help with the transition and tooling to automate some of the conversion: https://docs.python.org/3/library/ast.html https://docs.python.org/3/library/ast.html IIRC it wasn't there at the start though.
- marssaxman 4y ago
  What is a "codemod"?
  - deleted 4y ago
    [deleted]
  - charcircuit 4y ago
    A codemod is an automated refactor. Codemods are typically done by writing scripts that manipulate the AST of code in files or by writing a regex to find and replace. The name originates from an internal script at Facebook called codemod. It had a public open source fork if you are curious. https://github.com/facebookarchive/codemod https://github.com/facebookarchive/codemod

onphonenow 4y ago

The python maintainers went out of their way to break things - and were smug about it. That’s what it felt like to the community. A big part of 3 was Unicode strings. In 2 you could mark a string as Unicode with a u””. This would have been a great way to let libraries and code work with both 2 and 3. They banned this, but you had to still use b””. This was the attitude and just one example. The list of stuff that got hard / slow was long. Migration path was unnecessarily difficult as was compatible code. Somewhere around 3.4 it was like a light switch flipped and they started being more reasonable. ASCII handling improved (yes, not all internet protocol stuff is Unicode), they began making it easier to target 2/3 etc. but it was horrible to start. I’ve thankfully forgotten some of the details :)

bjt2n3904 4y ago
The era of "move fast and break things" was disastrous for a lot of things.
- acdha 4y ago
  12 years is moving fast?
simplotek 4y ago
> Migration path was unnecessarily difficult as was compatible code. I feel this take is outright wrong. Python provided it's 2to3 tool[1] which took care of the bulk of the work required to port Python2 code to Python3. The only code that was not supported was eggregious errors in the code that worked by coincidence, such as handling bytes as strings and vice-versa. Porting old code to Python3 is a breeze that consisted of running 2to3, run tests with python3, and if anything breaked then just touch up the code to get it to work. I know it because I personally ported half a dozen projects throughout the years. I'll go as far as to claim that most problems porting old python code to 3 were either upstream dependencies dragging their feet or internal human/organizational issues. [1] https://docs.python.org/3/library/2to3.html https://docs.python.org/3/library/2to3.html
- onphonenow 4y ago

amiga386 4y ago

The biggest problem is fundamental to Python and a lot of other scripting languages: no static typing. Running Python 2 code direct on Python 3? All looks good, until _somewhere_ you use a method of str that isn't on bytes, or vice versa, or something that was a list before is now an iterable and it blows up in your face... but _only_ if code execution reaches that line. So you have to test _everything_. And if you're a library/module, that means testing with _every user_. Secondly, it seems to me the Python team didn't want to make it possible to write code that works correctly on both Python 2 and 3 simultaneously. They wanted you to write code that only works on Python 3 and tell every single Python 2 user to get lost. Essentially, they wanted to co-opt you as their agent of change; burn your own reputation by abandoning your userbase (or worse, start proselytising Py3 at them) or double your own workload to maintain two codebases. Who wants that? It's fundamentally an attitude problem. A desire for there to be only one right way to do things, a change of mind of what the right way is, and kicking the chair out from anyone doing it the old right way; the only compromise being to kicking the chair very very slowly. In Java, you can still run code that uses Vector and Hashtable from 1996, they're not even deprecated. They're used by _very_ little code since generic List and Map APIs superceded them 2004, but they're still there.

xorcist 4y ago

The early Python 3 releases were abysmal, much slower than Python 2 even without considering the bugs. That alone was bad. But there was also real world problems, like how some syscall wrappers would not take bytestrings where they should have, so you literally could not open files unless you had a valid unicode representation of their file name. That's fun when all you want to do is some data processing. The core team was swamped with other work around the migration and the community found their response to real world problems lacking. They did things like declaring Python 2 dead long before 3 was usable. Then they refused to fix SSL problems with Python 2, which everyone desperately needed, and went out of their way to oppose a community-led release to fix those problems. It all ended well. There was another Python 2 release, and Python 3.4 came and fixed most problems people had. It was still slower, but performance picked up and by 3.7 it was just as good if not better. But it was a good ten years during which the response to real world issues could have been a lot better. All in all, the migration could have happened much smoother. They could have learned from other languages. But ten years is not unreasonable for backwards incompatible changes for a major programming language. The idea that the process could be accelerated by neglecting the old version does not work. In my personal opinion, the single thing that could have made the transition better for the community would have been the possibility to run Python 2 code in Python 3. It was discussed a lot before the transition and was deemed impossible because of conflicting data types. But it's not impossible, it's just (a lot of) work. It's another type conversion, and Python can already do those. The impossibility to link to old libraries made the conversion a flag day for most code bases, and those are really hard.