11 ms·

Usually yes. But almost all language transitions are easier than the Python 2->3 transition.

by dwheeler 6y ago

hprotagonist 6y ago
a decade's worth of grace period, on the other hand, is probably too generous.
- dwheeler 6y ago
  Not at all. It's only too generous if there is a low-cost way to incrementally transition to 3, as is normally the case with language updates. Python 2->3 was a poorly managed update that did not follow normal upgrade rules, so the "normal" rules don't apply. It is still often very extremely expensive to convert python2 to python3. Normally you could upgrade in small steps, maybe a file or library at a time, instead of changing everything simultaneously including all transitive dependencies. That problem continues to be denied, so python2 continues to be used. I say that as someone who has converted code from 2 to 3. Python3 is fine, it's the huge unnecessary transition cost that is not. It's gotten a little better, but not a lot better. This makes it worse. Now it's even harder to incrementally update, making it even harder to switch to 3.
  - michaelmrose 6y ago
    If you started only making new stuff in python 3 in 2010 2 years after python 3 came out how much python 2 would you have to convert? If of course you spent an extra decade producing obvious technical debt whose fault is it?
    - pishpash 6y ago
      What about code people wrote before 2010 that were perfectly fine? Are you going to have people rewrite research algorithms whose original authors have long graduated? Just because industry has a habit of rewriting the whole stack every five years on account of make-work job security doesn't make foundational scientific algorithms change.
      - philderbeast 6y ago
        if the code still has value being run in modern environments, then it probably needs to be maintained and updated anyway. if its not being used, then does it running with the latest tools even matter?
        lanstin 6y ago
        Not really. For example R ships with or depends on a lot of FORTRAN libraries. I doubt they have changed much at all in decades. There is no talk of a breaking FORTRAN language change that would require rewriting this perfectly FB ctional code with a stable interface.
      - michaelmrose 6y ago
        Couldn't you isolate the entire outdated environment this 2002-2009 code is expected to run in? Plenty of code written in this time frame is liable to depend on unsupported things by that point.
      - bastawhiz 6y ago
        Except the Python folks produced tools that did almost all of the work for you. 2to3 has worked for an overwhelming number of use cases. And in fact, most code requires very few changes to begin with. If this academic code isn't well understood or well tested, it's probably not as valuable as you might think.
        dwheeler 6y ago
        > 2to3 has worked for an overwhelming number of use cases. And in fact, most code requires very few changes to begin with. That is not at all my experience, nor is that experience of most other people I've ever talked to.
        __turbobrew__ 6y ago
        I ran 2to3 on 1 of about 200 modules at my workplace codebase and it took me about an hour to run 2to3 and fix up the leftovers in that module. Now do that for 199 other modules and it becomes much more work than “just run 2to3”.
    - dwheeler 6y ago
      > If you started only making new stuff in python 3 in 2010 2 years after python 3 came out how much python 2 would you have to convert? In many organizations there was never a time where they could start writing new code in Python 3. They needed to write code that was compatible with their existing python 2 code, and the only way to do that is to continue to write new code in python 2. Rinse, lather, repeat. This is why the failure to provide a gradual transition was so bad. When I write new code in Python I use python3, but that assumes that there are python3 modules available that I need. If you have infinite money this is not a problem. But I think we should be sympathetic to the people who do not have infinite money and have never been given a realistic upgrade path from 2 to 3. The 2to3 program is not a workable solution for many.
      - pseudalopex 6y ago
        You could write Python 2 that was compatible with Python 3 or at least would be easy to convert.
        user5994461 6y ago
        Not until the year 2015 and python 3.5 around the corner. That's when the interpreter finally got the minimum support required to make code compatible with both, and linters improved enough, and some libraries started being ported.
        pseudalopex 6y ago
        six 1.0 was in 2011.[1] 50% of the top 200 packages were compatible by the end of 2012.[2] And there were features you could use in 2008 to make the eventual conversion easier. [1] https://github.com/benjaminp/six/commits/1.0.0 https://github.com/benjaminp/six/commits/1.0.0 [2] https://python3wos.appspot.com/ https://python3wos.appspot.com/
        user5994461 6y ago
        Hence the ecosystem became usable around 2015 ;) It takes time between the initial release and enough fixes/additions to become usable, and the documentation/stackoverflow to cover it.
    - higeorge13 6y ago
      People don't like changes. I joined an AI company on 2018 which was building their whole prototype with python 2.7 since 2017. I had to spend 2 weeks and do the migration myself, do a merge request out of the blue and give it to their lead engineers, otherwise i am pretty sure they would have a meeting tomorrow 24/1/2021 to see how they are going to migrate.
  - joshuamorton 6y ago
    > Normally you could upgrade in small steps, maybe a file or library at a time, instead of changing everything simultaneously including all transitive dependencies. This was absolutely possible. Via the path of upgrading your code (or your dependencies code, in any arbitrary order) to be compatible with both py2 and py3 (via, say, six) and then once all code was compatible in either direction, flipping the switch.
    - fennecfoxen 6y ago
      It was always dangerously hard to validate that the whole codebase was actually compatible, and there wasn’t some path through the code that would make things break (in production on the customers’ site while handling all their most critical data). I can think of exactly one language who bungled the upgrade path worse and it’s Perl 6, which they finally renamed after 19 years of stringing people along like it would be the next big thing.
      - joshuamorton 6y ago
        I generally agree with your statement (the version flip is scary), but that's a different complaint than GGPs, which was more that you couldn't incrementally migrate your source files. The version flip to Java N+1 can be scary and cause issues too (and sure, it's perhaps less likely to cause issues, but don't worry, they both do!)
  - tgsovlerkhgsel 6y ago
    The main issue I ran into was string handling. Autoconversion handled most of the rest, but because byte-strings and regular strings were interchangeable in Py2, random places that weren't even touched by the autoconversion now suddenly have a "b'abc'" where they should have "abc", and fail in very non-obvious ways. Just some hack around that would probably already help.
    - eesmith 6y ago
      I had to change my public APIs in a backwards-incompatible way. I had a method something like obj.to_string("abc") to convert the object into a string in 'abc' format. I also supported things like obj.to_string("abc.gz") to get the record in gzip-compressed form. I also had a from_string() -> obj functions. You can see the problem. I had to change to_string() so it didn't allow compression (breaking backwards compatibility) and add a to_bytes() for that functionality, and add type dispatching in the from_string() code to support either byte or strings, with different code paths. And change all the open() calls to use "b", and to add type checks on user-passed-in file objects to insist on reading bytes (if not isinstance(user_file.read(0), bytes) raise "Must be open in binary mode") because all of the underlying parsers are in C and the needless decode/encode step adds overhead. Oh, and re-write the C extension so it handles both Python 2 and Python 3. That was a really boring 6 weeks.
- ed25519FUUU 6y ago
  Have you stopped to ask why the transition was so bad?
smitty1e 6y ago
Whereas the Perl 5.8 upgrade pretty much killed the language.
mumblemumble 6y ago
At this point, I swear, the main source of difficulty in Python 2->3 is all the foot-dragging in migrating to Python 3. I wouldn't be at all surprised to discover that vastly more Python 2 code was written in the past twelve years than in the preceding eight.
- MattGaiser 6y ago
  A friend of mine uses Python in his lab. Tons of them were writing Python 2 code when they were learning it two years ago (Feb 2019). So they started from scratch with it two years ago.
- josefx 6y ago
  > the main source of difficulty in Python 2->3 is all the foot-dragging in migrating to Python 3. My main difficulty was having targets that ran python 2 and wouldn't get an official python 3 package unless a miracle happened. Meanwhile most systems supporting python 3 also had a python 2 package. Ended up porting a few scripts to a statically linked c++ binary, those should keep on working until we get 128 bit systems that drop support for 64 bit binaries.
  - busterarm 6y ago
    You learned an important lesson about distributing software and business risk. In my experiences trying to deploy software at scale (10^4~5), statically linked binaries are the only acceptable form of software.
    - yjftsjthsd-h 6y ago
      If you don't have a pipeline to update those static binaries, you're just giving yourself a different form of headache 5 years from now when you need to apply bug fix.
      - lanstin 6y ago
        With static linking you mostly have all the pieces updatable independently. Can move to new distro and keep the binaries. Can update some binaries with a bug fix to some parser library without releasing other binaries. Etc. except for glibc and stuff like 64-bit inode values.
philderbeast 6y ago
Python 3 was released in 2008, it had 12 years of legacy support before going EOL, and then an additional 12 months of support after that. If 13 years isn't enough warning to migrate your use case, no amount of warning will help you. At some point you have to stop saying its to hard to migrate.
- pishpash 6y ago
  Academia doesn't have unlimited budget to do migrations that don't solve any pressing problems. I guess now there is a pressing problem, but more likely people will maintain the old version of pip locally.
  - philderbeast 6y ago
    Then they are asking the wrong question, instead of asking do I have to upgrade, they need to be asking why shouldn't I upgrade. Keeping current with industry is the only way they can stay relevant in the modern world, particularly in IT and related fields where anyone with a computer at home can do the same research as someone at a university.
  - meekrohprocess 6y ago
    Yeah, really. Graduate students aren't cheap! Oh wait, we pay computing grad students literally 1/10 of their potential salary ($10Ks vs $100Ks). Why could there possibly be a shortage? -_- More importantly, it shouldn't be prohibitively expensive to port academic Python code.
    - Wowfunhappy 6y ago
      Graduate students are there to do research, not update old existing codebases. That's a part of the bargain, part in parcel with the low pay.
      - Kalium 6y ago
        From limited exposure, these codebases are often in truly awful shape. They've endured years or decades of having been hacked up just enough to finish someone's thesis or dissertation with no concern at all towards maintenance. It should be no surprise that such a poorly engineered process produces awful results.
melling 6y ago
Let me quote myself from 7 years ago. “ It's kind of disappointing that developers aren't self-aware enough to understand that they are essentially screwing themselves by not moving quickly in dropping "legacy" support. "Python 3 will be available in 2008 but we understand that you won't use it for at least 5 years." There's an entire class of developers who won't upgrade until they absolutely have to. There's also a class of administrator that won't upgrade their current PC browser from IE6-IE8 until they absolutely have to. Developers are basically screwing themselves by not drawing a firm line.” https://news.ycombinator.com/item?id=7372913 https://news.ycombinator.com/item?id=7372913
- Wowfunhappy 6y ago
  Or, everyone leaves your platform because they can't keep up with the breaking changes.
  - adamc 6y ago
    Less a problem with languages, in my experience, than libraries. Django updates are a pain in the rear.
    - deleted 6y ago
      [deleted]
    - Wowfunhappy 6y ago
      idk. I think there's an alternate universe somewhere in which Python just got hard forked by the community, and the python foundation lost its influence over the direction of the language. The split today would be even larger, or Python 3 would have become an intellectual curiosity rarely used in the real world.
- adamc 6y ago
  There isn't really a good solution. Forced migrations always create a decision point, at which point people may gather their resentments and move to something else. But if you don't eventually force a change, you have to live with all of the language's historical mistakes. My shop was interested in migrating long ago, but we had to wait until languages and frameworks (such as Django) supported 3. On projects where that wasn't a risk, we moved to Python around 2014 or so. But: I totally get that for some shops, the feasibility of migrating was low.

Usually yes. But almost all language transitions are easier than the Python 2->3 transition.

hprotagonist 6y ago

a decade's worth of grace period, on the other hand, is probably too generous.

dwheeler 6y ago
Not at all. It's only too generous if there is a low-cost way to incrementally transition to 3, as is normally the case with language updates. Python 2->3 was a poorly managed update that did not follow normal upgrade rules, so the "normal" rules don't apply. It is still often very extremely expensive to convert python2 to python3. Normally you could upgrade in small steps, maybe a file or library at a time, instead of changing everything simultaneously including all transitive dependencies. That problem continues to be denied, so python2 continues to be used. I say that as someone who has converted code from 2 to 3. Python3 is fine, it's the huge unnecessary transition cost that is not. It's gotten a little better, but not a lot better. This makes it worse. Now it's even harder to incrementally update, making it even harder to switch to 3.
- michaelmrose 6y ago
  If you started only making new stuff in python 3 in 2010 2 years after python 3 came out how much python 2 would you have to convert? If of course you spent an extra decade producing obvious technical debt whose fault is it?
  - pishpash 6y ago
    What about code people wrote before 2010 that were perfectly fine? Are you going to have people rewrite research algorithms whose original authors have long graduated? Just because industry has a habit of rewriting the whole stack every five years on account of make-work job security doesn't make foundational scientific algorithms change.
    - philderbeast 6y ago
      if the code still has value being run in modern environments, then it probably needs to be maintained and updated anyway. if its not being used, then does it running with the latest tools even matter?
      - lanstin 6y ago
        Not really. For example R ships with or depends on a lot of FORTRAN libraries. I doubt they have changed much at all in decades. There is no talk of a breaking FORTRAN language change that would require rewriting this perfectly FB ctional code with a stable interface.
    - michaelmrose 6y ago
      Couldn't you isolate the entire outdated environment this 2002-2009 code is expected to run in? Plenty of code written in this time frame is liable to depend on unsupported things by that point.
    - bastawhiz 6y ago
      Except the Python folks produced tools that did almost all of the work for you. 2to3 has worked for an overwhelming number of use cases. And in fact, most code requires very few changes to begin with. If this academic code isn't well understood or well tested, it's probably not as valuable as you might think.
      - dwheeler 6y ago
        > 2to3 has worked for an overwhelming number of use cases. And in fact, most code requires very few changes to begin with. That is not at all my experience, nor is that experience of most other people I've ever talked to.
        __turbobrew__ 6y ago
        I ran 2to3 on 1 of about 200 modules at my workplace codebase and it took me about an hour to run 2to3 and fix up the leftovers in that module. Now do that for 199 other modules and it becomes much more work than “just run 2to3”.
  - dwheeler 6y ago
    > If you started only making new stuff in python 3 in 2010 2 years after python 3 came out how much python 2 would you have to convert? In many organizations there was never a time where they could start writing new code in Python 3. They needed to write code that was compatible with their existing python 2 code, and the only way to do that is to continue to write new code in python 2. Rinse, lather, repeat. This is why the failure to provide a gradual transition was so bad. When I write new code in Python I use python3, but that assumes that there are python3 modules available that I need. If you have infinite money this is not a problem. But I think we should be sympathetic to the people who do not have infinite money and have never been given a realistic upgrade path from 2 to 3. The 2to3 program is not a workable solution for many.
    - pseudalopex 6y ago
      You could write Python 2 that was compatible with Python 3 or at least would be easy to convert.
      - user5994461 6y ago
        Not until the year 2015 and python 3.5 around the corner. That's when the interpreter finally got the minimum support required to make code compatible with both, and linters improved enough, and some libraries started being ported.
        pseudalopex 6y ago
        six 1.0 was in 2011.[1] 50% of the top 200 packages were compatible by the end of 2012.[2] And there were features you could use in 2008 to make the eventual conversion easier. [1] https://github.com/benjaminp/six/commits/1.0.0 https://github.com/benjaminp/six/commits/1.0.0 [2] https://python3wos.appspot.com/ https://python3wos.appspot.com/
        user5994461 6y ago
        Hence the ecosystem became usable around 2015 ;) It takes time between the initial release and enough fixes/additions to become usable, and the documentation/stackoverflow to cover it.
  - higeorge13 6y ago
    People don't like changes. I joined an AI company on 2018 which was building their whole prototype with python 2.7 since 2017. I had to spend 2 weeks and do the migration myself, do a merge request out of the blue and give it to their lead engineers, otherwise i am pretty sure they would have a meeting tomorrow 24/1/2021 to see how they are going to migrate.
- joshuamorton 6y ago
  > Normally you could upgrade in small steps, maybe a file or library at a time, instead of changing everything simultaneously including all transitive dependencies. This was absolutely possible. Via the path of upgrading your code (or your dependencies code, in any arbitrary order) to be compatible with both py2 and py3 (via, say, six) and then once all code was compatible in either direction, flipping the switch.
  - fennecfoxen 6y ago
    It was always dangerously hard to validate that the whole codebase was actually compatible, and there wasn’t some path through the code that would make things break (in production on the customers’ site while handling all their most critical data). I can think of exactly one language who bungled the upgrade path worse and it’s Perl 6, which they finally renamed after 19 years of stringing people along like it would be the next big thing.
    - joshuamorton 6y ago
      I generally agree with your statement (the version flip is scary), but that's a different complaint than GGPs, which was more that you couldn't incrementally migrate your source files. The version flip to Java N+1 can be scary and cause issues too (and sure, it's perhaps less likely to cause issues, but don't worry, they both do!)
- tgsovlerkhgsel 6y ago
  The main issue I ran into was string handling. Autoconversion handled most of the rest, but because byte-strings and regular strings were interchangeable in Py2, random places that weren't even touched by the autoconversion now suddenly have a "b'abc'" where they should have "abc", and fail in very non-obvious ways. Just some hack around that would probably already help.
  - eesmith 6y ago
    I had to change my public APIs in a backwards-incompatible way. I had a method something like obj.to_string("abc") to convert the object into a string in 'abc' format. I also supported things like obj.to_string("abc.gz") to get the record in gzip-compressed form. I also had a from_string() -> obj functions. You can see the problem. I had to change to_string() so it didn't allow compression (breaking backwards compatibility) and add a to_bytes() for that functionality, and add type dispatching in the from_string() code to support either byte or strings, with different code paths. And change all the open() calls to use "b", and to add type checks on user-passed-in file objects to insist on reading bytes (if not isinstance(user_file.read(0), bytes) raise "Must be open in binary mode") because all of the underlying parsers are in C and the needless decode/encode step adds overhead. Oh, and re-write the C extension so it handles both Python 2 and Python 3. That was a really boring 6 weeks.
ed25519FUUU 6y ago
Have you stopped to ask why the transition was so bad?

smitty1e 6y ago

Whereas the Perl 5.8 upgrade pretty much killed the language.

mumblemumble 6y ago

At this point, I swear, the main source of difficulty in Python 2->3 is all the foot-dragging in migrating to Python 3. I wouldn't be at all surprised to discover that vastly more Python 2 code was written in the past twelve years than in the preceding eight.

MattGaiser 6y ago
A friend of mine uses Python in his lab. Tons of them were writing Python 2 code when they were learning it two years ago (Feb 2019). So they started from scratch with it two years ago.
josefx 6y ago
> the main source of difficulty in Python 2->3 is all the foot-dragging in migrating to Python 3. My main difficulty was having targets that ran python 2 and wouldn't get an official python 3 package unless a miracle happened. Meanwhile most systems supporting python 3 also had a python 2 package. Ended up porting a few scripts to a statically linked c++ binary, those should keep on working until we get 128 bit systems that drop support for 64 bit binaries.
- busterarm 6y ago
  You learned an important lesson about distributing software and business risk. In my experiences trying to deploy software at scale (10^4~5), statically linked binaries are the only acceptable form of software.
  - yjftsjthsd-h 6y ago
    If you don't have a pipeline to update those static binaries, you're just giving yourself a different form of headache 5 years from now when you need to apply bug fix.
    - lanstin 6y ago
      With static linking you mostly have all the pieces updatable independently. Can move to new distro and keep the binaries. Can update some binaries with a bug fix to some parser library without releasing other binaries. Etc. except for glibc and stuff like 64-bit inode values.

philderbeast 6y ago

Python 3 was released in 2008, it had 12 years of legacy support before going EOL, and then an additional 12 months of support after that. If 13 years isn't enough warning to migrate your use case, no amount of warning will help you. At some point you have to stop saying its to hard to migrate.

pishpash 6y ago
Academia doesn't have unlimited budget to do migrations that don't solve any pressing problems. I guess now there is a pressing problem, but more likely people will maintain the old version of pip locally.
- philderbeast 6y ago
  Then they are asking the wrong question, instead of asking do I have to upgrade, they need to be asking why shouldn't I upgrade. Keeping current with industry is the only way they can stay relevant in the modern world, particularly in IT and related fields where anyone with a computer at home can do the same research as someone at a university.
- meekrohprocess 6y ago
  Yeah, really. Graduate students aren't cheap! Oh wait, we pay computing grad students literally 1/10 of their potential salary ($10Ks vs $100Ks). Why could there possibly be a shortage? -_- More importantly, it shouldn't be prohibitively expensive to port academic Python code.
  - Wowfunhappy 6y ago
    Graduate students are there to do research, not update old existing codebases. That's a part of the bargain, part in parcel with the low pay.
    - Kalium 6y ago
      From limited exposure, these codebases are often in truly awful shape. They've endured years or decades of having been hacked up just enough to finish someone's thesis or dissertation with no concern at all towards maintenance. It should be no surprise that such a poorly engineered process produces awful results.

melling 6y ago

Let me quote myself from 7 years ago. “ It's kind of disappointing that developers aren't self-aware enough to understand that they are essentially screwing themselves by not moving quickly in dropping "legacy" support. "Python 3 will be available in 2008 but we understand that you won't use it for at least 5 years." There's an entire class of developers who won't upgrade until they absolutely have to. There's also a class of administrator that won't upgrade their current PC browser from IE6-IE8 until they absolutely have to. Developers are basically screwing themselves by not drawing a firm line.” https://news.ycombinator.com/item?id=7372913 https://news.ycombinator.com/item?id=7372913

Wowfunhappy 6y ago
Or, everyone leaves your platform because they can't keep up with the breaking changes.
- adamc 6y ago
  Less a problem with languages, in my experience, than libraries. Django updates are a pain in the rear.
  - deleted 6y ago
    [deleted]
  - Wowfunhappy 6y ago
    idk. I think there's an alternate universe somewhere in which Python just got hard forked by the community, and the python foundation lost its influence over the direction of the language. The split today would be even larger, or Python 3 would have become an intellectual curiosity rarely used in the real world.
adamc 6y ago
There isn't really a good solution. Forced migrations always create a decision point, at which point people may gather their resentments and move to something else. But if you don't eventually force a change, you have to live with all of the language's historical mistakes. My shop was interested in migrating long ago, but we had to wait until languages and frameworks (such as Django) supported 3. On projects where that wasn't a risk, we moved to Python around 2014 or so. But: I totally get that for some shops, the feasibility of migrating was low.