7 ms·

Pandas dropping Python 2 support

122 points by ptype 8y ago

gammateam 8y ago
Amazing, everyone dropping python2 should have been 2010s headline but hey whats another decade
- orf 8y ago
  I'm not sure the packaging ecosystem, the project messaging/marketing or the interpreter itself was ready for the transition then. Which is unfortunate. But hey, all good things come to those who wait. And wait. And wait.
  - sephoric 8y ago
    Sometimes all that's needed is a little pressure and motivation mixed with morale from strong leadership to encourage people to spent a day or so upgrading and testing. Maybe not in this case though. That might not have been enough here, who knows.
    - AnonymousPlanet 8y ago
      This kind of smug armchair analysis is great as long as all your projects are just a couple of thousand lines. Try updating in "a day or so" when you have millions of lines of Python 2 code. And your code runs at hundreds of customer sites. "Two years or so" is much more realistic. If and only if you have the manpower.
      - marcosdumay 8y ago
        Well, I shrug imagining what millions of lines of Python looks like. I can't think of any open Python project with that size, and I can't imagine an organization maintaining one. And I can easily agree that such project won't migrate from Python 2 to 3 in a year. It won't migrate in a decade either, or in a century.
        hermitdev 8y ago
        At a previous employer, I left behind maybe not millions of lines of Python 2 code, but certainly somewhere in the hundreds of thousands range. There was also a fair amount of reliance on 3rd party libs for databases, IBM MQ and tibrv that had native components. We also had our own Python libs that had native components written using Boost Python. Large Python projects are possible, but if you don't have good discipline and organization, it can quickly become a mess (probably true regardless of the language). I'm not sure how much effort it'd be to migrate all of that to Python 3, but I'm sure it's not trivial. But, I also don't think it'd be a multi year effort. It also wasn't a big monolithic project, so a migration could be done piecemeal.
        eesmith 8y ago
        You made me curious so I looked around. I couldn't easily find an open Python project with 1M+ LOC. Python 3.7 has 542,292 lines of Python, according to David A. Wheeler's 'SLOCCount' . And another 418,581 lines of C. NumPy has 159,208 lines of C and 113,842 of Python. SciPy has 174,580 lines of Python, 83,155 of C, and 79,514 of Fortran. Pandas has 225,417 lines of Python. Put them all together and that's 1M lines of Python to run the standard Python data stack.
        AnonymousPlanet 8y ago
        I wasn't talking hypothetical, I was talking out of my own experience. Yes, these codebases exist and, yes, maintenance isn't easy, especially because of dynamic typing.
        jakevn 8y ago
        OpenStack has millions of lines of Python, I believe.
      - ilayn 8y ago
        I used to think along those lines but I don't buy this argument anymore. Many impossible-looking companies just switched pretty seamlessly Instagram, Dropbox etc being among the big ones. Besides, you never hear about what is difficult about the migration as opposed to why it might be difficult because a lot of lines. I have been hearing this since a few years instead of repeating the same thing they would have ported by now should they even bothered. Here are all the milestones https://python3statement.org/ https://python3statement.org/
        celias 8y ago
        There is a discussion about Instagram's approach to migrating to python3 around the 34 minute mark in this talkpython podcast https://talkpython.fm/episodes/show/155/practical-steps-for-moving-to-python-3 https://talkpython.fm/episodes/show/155/practical-steps-for-... Lisa Guo's presentation starts around the 13 minute mark in the 2017 keynote linked to on that page. Here's a blog post about Dropbox migrating from python 2 to 3 https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever/ https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one...
        pseudalopex 8y ago
        Facebook and Dropbox took years to switch. Instagram took at least 10 months. (I think that was just the big push.) They switched after the Python developers added compatibility features in 2011 and 2012 and libraries had time to catch up.
- rogerbinns 8y ago
  You didn't hear much from the people using Python 2 to get things done. Here is what happened in that world: * there were no syntax changes, no new keywords, no deprecations, no coloured functions (http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...), no major changes in the standard library etc * some sore points (eg subprocess & threads) had backported modules with fixes * python 2.7 continued to be supported by major third party libraries, and they tended to avoid wholesale changes * binary compatibility and testing are easy (I have a python binary module compiled on 32 bit ARM in 2014 that still loads without recompilation today) ie python 2 is more than sufficient to solve real world problems, and in the last decade has been stable, debugged, and predictable. You generally don't hear from folks doing this! Much like C++, Python 3 still has the problem of which subset of language features to use since they affect minimum Python 3.x version, interaction with libraries etc. The Python 2 -> 3 transition is going to be a case study for a long time. All the intentions and planning seemed right, yet here we are a decade later. There are difficult trade offs between new productivity, stability and predictability, and giving things time to mature, with space to make mistakes. As an example Rust seems to be applying the lessons learned.
  - throwaway12iii 8y ago
    What is wrong with LTS releases? That's what python 2.7 is.
  - orf 8y ago
    > * binary compatibility and testing are easy (I have a python binary module compiled on 32 bit ARM in 2014 that still loads without recompilation today) I'm confused, are you trying to say a module compiled for Python 2.7 still works with Python 2.7?
    - rogerbinns 8y ago
      I am saying exactly that. Note that it still works across compiler upgrades, libc updates, OS upgrades, python itself updates etc. By contrast if you compile a module against Python 3.4, it won't even work the same day against Python 3.5. ie the ever changing Python 3.x is more busy work than Python 2.7.
abakus 8y ago
Time to move forward. Python 2 is so 2010.
amypinka 8y ago
There was a call asking for volunteers to continue to support 2.7. There's something like 60K lines of Python code in the project so perhaps coding standards keeping the 3.x-specific syntax to a minimum means the project can continue to support a wide installation base without a great deal of overhead for any would-be 2.7 maintainer.
- orf 8y ago
  Sacrificing the awesome 3.x features (f-strings, typing, metaclasses, object model improvements) to hold onto a deprecated and soon-to-be obsolete version of Python seems bad to me. IMO make a clean cut else it will never end.
  - beagle3 8y ago
    Python 1 already had metaclasses (I remebmer it as "the Don Beaudry hack" or "the Don Beaudry hook", depended on who you asked) Are Python 3 metaclasses a significant change? [0] https://www.python.org/doc/essays/metaclasses/ https://www.python.org/doc/essays/metaclasses/
    - orf 8y ago
      There are a number of improvements, including how you define them and the things you can do with them. Some of these tie in with other language features, e.g typing.NamedTuple[1]. There is also PEP 487[2], which means you can do more without needing a metaclass. 1. https://docs.python.org/3/library/typing.html#typing.NamedTuple https://docs.python.org/3/library/typing.html#typing.NamedTu... 2. https://www.python.org/dev/peps/pep-0487/ https://www.python.org/dev/peps/pep-0487/
- oliwarner 8y ago
  This isn't sustainable. These new language features make for cleaner, leaner, even faster codebases. Python 2 dinosaurs will need to come to with a Babel-style project to transliterate modern syntax —3to2, if you will— and rebase their code from that. It's going to be hell to maintain.
beams_of_light 8y ago
This is painful for Debian users.
- evgen 8y ago
  apt-get install python3 Problem solved.

122 points by ptype 8y ago

gammateam 8y ago

Amazing, everyone dropping python2 should have been 2010s headline but hey whats another decade

orf 8y ago
I'm not sure the packaging ecosystem, the project messaging/marketing or the interpreter itself was ready for the transition then. Which is unfortunate. But hey, all good things come to those who wait. And wait. And wait.
- sephoric 8y ago
  Sometimes all that's needed is a little pressure and motivation mixed with morale from strong leadership to encourage people to spent a day or so upgrading and testing. Maybe not in this case though. That might not have been enough here, who knows.
  - AnonymousPlanet 8y ago
    This kind of smug armchair analysis is great as long as all your projects are just a couple of thousand lines. Try updating in "a day or so" when you have millions of lines of Python 2 code. And your code runs at hundreds of customer sites. "Two years or so" is much more realistic. If and only if you have the manpower.
    - marcosdumay 8y ago
      Well, I shrug imagining what millions of lines of Python looks like. I can't think of any open Python project with that size, and I can't imagine an organization maintaining one. And I can easily agree that such project won't migrate from Python 2 to 3 in a year. It won't migrate in a decade either, or in a century.
      - hermitdev 8y ago
        At a previous employer, I left behind maybe not millions of lines of Python 2 code, but certainly somewhere in the hundreds of thousands range. There was also a fair amount of reliance on 3rd party libs for databases, IBM MQ and tibrv that had native components. We also had our own Python libs that had native components written using Boost Python. Large Python projects are possible, but if you don't have good discipline and organization, it can quickly become a mess (probably true regardless of the language). I'm not sure how much effort it'd be to migrate all of that to Python 3, but I'm sure it's not trivial. But, I also don't think it'd be a multi year effort. It also wasn't a big monolithic project, so a migration could be done piecemeal.
      - eesmith 8y ago
        You made me curious so I looked around. I couldn't easily find an open Python project with 1M+ LOC. Python 3.7 has 542,292 lines of Python, according to David A. Wheeler's 'SLOCCount' . And another 418,581 lines of C. NumPy has 159,208 lines of C and 113,842 of Python. SciPy has 174,580 lines of Python, 83,155 of C, and 79,514 of Fortran. Pandas has 225,417 lines of Python. Put them all together and that's 1M lines of Python to run the standard Python data stack.
      - AnonymousPlanet 8y ago
        I wasn't talking hypothetical, I was talking out of my own experience. Yes, these codebases exist and, yes, maintenance isn't easy, especially because of dynamic typing.
      - jakevn 8y ago
        OpenStack has millions of lines of Python, I believe.
    - ilayn 8y ago
      I used to think along those lines but I don't buy this argument anymore. Many impossible-looking companies just switched pretty seamlessly Instagram, Dropbox etc being among the big ones. Besides, you never hear about what is difficult about the migration as opposed to why it might be difficult because a lot of lines. I have been hearing this since a few years instead of repeating the same thing they would have ported by now should they even bothered. Here are all the milestones https://python3statement.org/ https://python3statement.org/
      - celias 8y ago
        There is a discussion about Instagram's approach to migrating to python3 around the 34 minute mark in this talkpython podcast https://talkpython.fm/episodes/show/155/practical-steps-for-moving-to-python-3 https://talkpython.fm/episodes/show/155/practical-steps-for-... Lisa Guo's presentation starts around the 13 minute mark in the 2017 keynote linked to on that page. Here's a blog post about Dropbox migrating from python 2 to 3 https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one-of-the-largest-python-3-migrations-ever/ https://blogs.dropbox.com/tech/2018/09/how-we-rolled-out-one...
      - pseudalopex 8y ago
        Facebook and Dropbox took years to switch. Instagram took at least 10 months. (I think that was just the big push.) They switched after the Python developers added compatibility features in 2011 and 2012 and libraries had time to catch up.
rogerbinns 8y ago
You didn't hear much from the people using Python 2 to get things done. Here is what happened in that world: * there were no syntax changes, no new keywords, no deprecations, no coloured functions (http://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/ http://journal.stuffwithstuff.com/2015/02/01/what-color-is-y...), no major changes in the standard library etc * some sore points (eg subprocess & threads) had backported modules with fixes * python 2.7 continued to be supported by major third party libraries, and they tended to avoid wholesale changes * binary compatibility and testing are easy (I have a python binary module compiled on 32 bit ARM in 2014 that still loads without recompilation today) ie python 2 is more than sufficient to solve real world problems, and in the last decade has been stable, debugged, and predictable. You generally don't hear from folks doing this! Much like C++, Python 3 still has the problem of which subset of language features to use since they affect minimum Python 3.x version, interaction with libraries etc. The Python 2 -> 3 transition is going to be a case study for a long time. All the intentions and planning seemed right, yet here we are a decade later. There are difficult trade offs between new productivity, stability and predictability, and giving things time to mature, with space to make mistakes. As an example Rust seems to be applying the lessons learned.
- throwaway12iii 8y ago
  What is wrong with LTS releases? That's what python 2.7 is.
- orf 8y ago
  > * binary compatibility and testing are easy (I have a python binary module compiled on 32 bit ARM in 2014 that still loads without recompilation today) I'm confused, are you trying to say a module compiled for Python 2.7 still works with Python 2.7?
  - rogerbinns 8y ago
    I am saying exactly that. Note that it still works across compiler upgrades, libc updates, OS upgrades, python itself updates etc. By contrast if you compile a module against Python 3.4, it won't even work the same day against Python 3.5. ie the ever changing Python 3.x is more busy work than Python 2.7.

abakus 8y ago

Time to move forward. Python 2 is so 2010.

amypinka 8y ago

There was a call asking for volunteers to continue to support 2.7. There's something like 60K lines of Python code in the project so perhaps coding standards keeping the 3.x-specific syntax to a minimum means the project can continue to support a wide installation base without a great deal of overhead for any would-be 2.7 maintainer.

orf 8y ago
Sacrificing the awesome 3.x features (f-strings, typing, metaclasses, object model improvements) to hold onto a deprecated and soon-to-be obsolete version of Python seems bad to me. IMO make a clean cut else it will never end.
- beagle3 8y ago
  Python 1 already had metaclasses (I remebmer it as "the Don Beaudry hack" or "the Don Beaudry hook", depended on who you asked) Are Python 3 metaclasses a significant change? [0] https://www.python.org/doc/essays/metaclasses/ https://www.python.org/doc/essays/metaclasses/
  - orf 8y ago
    There are a number of improvements, including how you define them and the things you can do with them. Some of these tie in with other language features, e.g typing.NamedTuple[1]. There is also PEP 487[2], which means you can do more without needing a metaclass. 1. https://docs.python.org/3/library/typing.html#typing.NamedTuple https://docs.python.org/3/library/typing.html#typing.NamedTu... 2. https://www.python.org/dev/peps/pep-0487/ https://www.python.org/dev/peps/pep-0487/
oliwarner 8y ago
This isn't sustainable. These new language features make for cleaner, leaner, even faster codebases. Python 2 dinosaurs will need to come to with a Babel-style project to transliterate modern syntax —3to2, if you will— and rebase their code from that. It's going to be hell to maintain.

beams_of_light 8y ago

This is painful for Debian users.

evgen 8y ago
apt-get install python3 Problem solved.