6 ms·

If one is on the Python 3 train, is there any reason to not always be on the latest release? Is it better about breaking changes than the 2 to 3 adventure was?

sprior 4y ago
Sometimes you're in an organization that doesn't upgrade things right away and it can be difficult to tell which features you can use and which you don't have yet.
blopker 4y ago
I think it depends on how large your codebase is and how many dependencies it has. The first issue is that any new 3.x release is going to have bugs introduced by new features, and possibly regressions in old ones. This could cause your current code to behave differently, as well as cause issues if someone immediately uses a new feature. The other issue is the more dependencies you have, the more likely one of them will not work with the newest version. Usually these things get sorted out pretty quickly, but it's frustrating to get a bunch of errors. I typically wait until at least the first bug fix release (3.11.1 for example) to try out a new Python version. These often have most of the common issues fixed, and give enough time for all the popular dependencies to catch up. This is from my experience maintaining a million+ line codebase that started at Python 2.6 and was updated to 3.8 as new versions came out.
reuven 4y ago
There are very, very few breaking changes within Python 3. If you're an individual user, then upgrading to the latest version is almost certainly a good idea. But there are lots of companies that release software using Python. To ensure a stable production environment, they upgrade more slowly, or on a schedule that's distinct from the Python release cycle. Also, companies often don't just let people upgrade things themselves. The IT department schedules a company-wide Python upgrade, for example, testing the effects beforehand. I have several training clients still using 3.7 and 3.8, not because things would break with 3.10, but because they haven't fully vetted the newest version, and they're being super conservative. I do wonder if companies will be a bit faster to upgrade over the coming years, given that execution speed is expected to improve rather dramatically. Upgrading won't be a matter of some nice-to-have features, but rather of real savings in system usage.
- smitty1e 4y ago
  > There are very, very few breaking changes within Python 3. But will structural pattern matching be back-ported, though? Certainly an incompatibility, if not a breaking change per se.
  - kdmccormick 4y ago
    GP means there are very few backwards-incompatible changes in new releases. There are plenty of forwards-incompatible ones, though.
  - fnord123 4y ago
    How is pattern matching a breaking change? If your code doesn't have it how would it break?
    - nedbat 4y ago
      Some people might mistakenly assume that "match" and "case" are now hard keywords, which would of course break lots of code. But they are soft keywords, so there's no issue.
david2ndaccount 4y ago
There are breaking changes in the C api occasionally, so that can be a good reason to not upgrade. Sometimes an extension needs to use the unstable API.
theamk 4y ago
If using python for large project, sure, use latest version. If writing small tools, consder limiting yourself to system python and its stdlib / system-packaged libraries. A script which ypu can download as a single file, immediately run, and immediately make some changes in, all without worrying about venvs or other setup, is a beautiful thing. This works especially well if you are writing internal tools and your company has standarts like "Everyone is on Ubuntu 20.04 or 22.04"
heavyset_go 4y ago
Sometimes. If you're relying on a toolchain that's parsing and analyzing Python code, things like compilers and mypy will sometimes lag behind new syntax, bytecode and C API changes. Until recently mypy didn't have full support for match statements, and its corresponding compiler, mypyc, still doesn't. Same goes, or went, for full support for match statements in other compilers, as well.
vorhemus 4y ago
Some popular packages are Python version dependent. For example, I was using Python 3.10 early and no Python wheels for 3.10 were released for PyTorch back then. The only thing you can do in this case is to either wair for the publisher to provide these packages or use a lower version Python interpreter.
- itsgrimetime 4y ago
  The 3.10 interpreter would still run 3.9-compatible Python code though, right?
  - faho 4y ago
    For python code, yeah, most of it. Python does sometimes have backwards-incompatible changes, e.g. for 3.10 they removed a bunch of stdlib modules and methods, like the "formatter" and "parser" module [0]. So if you used those, your code wouldn't work in 3.10. But the main reason to wait for wheels (which is pythonese for "pre-built packages") is if they use native code (like C or rust) and you would have to compile them yourself otherwise, which increases installation time quite a bit. (this was also the reason why Alpine was a bad choice for python containers for a long time because it uses musl and there were only wheels for glibc available. AFAIK musl wheels exist now so that isn't relevant anymore) [0]: https://docs.python.org/3/whatsnew/3.10.html#removed https://docs.python.org/3/whatsnew/3.10.html#removed
    - diarrhea 4y ago
      I always wondered what’s keeping packages back from having wheels ready the day a release drops, or way before even? I imagine often it’s just another parameter in a CI somewhere, where things mostly “just work” because backward-incompatible breakages have become much rarer.
      - singhrac 4y ago
        It's usually a question of having the resources (usually developers to debug issues). IIRC pretty much all Windows wheels are maintained by one guy, Christoph Gohlke, he was a godsend when I worked on a Windows laptop. I owe that guy many beers.
0x008 4y ago
Yea, if you use standard docker images o you want to use the python version they come with which is usually 3.8 (Ubuntu 20.04) or 3.6 (Ubuntu 18.04)
- arjvik 4y ago
  Why not use the library/python image with the specific version over library/Ubuntu?
  - 0x008 4y ago
    I don’t understand your question, but to illustrate the point: We work often with 3rd party libraries and images provided by those third parties. These are often based straight up on Ubuntu. Nvidia CUDA image is an example of this. Then you are stuck with their python version. And it’s not as easy as installing your own python version into the image because python dependencies inside those 3rd party images might be installed using OS package managers and not pip so then they are installed to the wrong python version etc. It’s a rabbit hole. The problem is that at some point Debian/ubuntu decided to repackage python dependencies as apt packages.
OJFord 4y ago
There have been breaking changes, I don't think anything too difficult to overcome, but it's easy to pin it intending to fix later, and fall behind. (e.g. 3.9-3.10 something to do with dataclasses iirc)
lynndotpy 4y ago
I find Conda packages often have a maximum version that's a bit out of date, sometimes several versions behind. I'm sure using 3.10 instead of 3.9 would be fine for most packages, but I don't know if it can be "forced"
- darkarmani 4y ago
  Compiled packages have ABI restrictions. Even conda-forge puts major version constraints on their packages at build time (ie: built on 3.9: >=3.9,<3.10.0a).
ramraj07 4y ago
Unfortunately some python packages are not reliably able to move across versions. I’ve had issues with sqlalchemy and celery before. Let’s not get started with the python packages from snowflake.
dragonwriter 4y ago
If you are dependent on binary packages rather than building extensions from source (more common on Windows), there may not be prebuilt packages for some libraries for the newest version.
Steltek 4y ago
If you're writing internal tools for your teammates, don't use the latest release. Your coworkers will almost certainly be on something older and you'll get super weird bugs. And you'll get the most bizarre sounding pushback when you tell them to install a newer version of Python.
digisign 4y ago
There are many little incompatibilities over 3.x. Especially at 3.10/11 where deprecations are finally being removed. Few big changes, as from 2.x. Wheels typically lag however. Release[-2] or Release[-3] are good choices. Definitely upgrade from [-4] or lower.
InfoSecErik 4y ago
For my Python things, I aimed to be compatible with the oldest supported Python for people on older OS's or who aren't constantly updating their Python installations. That way my stuff was still accessible to them.

sprior 4y ago

Sometimes you're in an organization that doesn't upgrade things right away and it can be difficult to tell which features you can use and which you don't have yet.

blopker 4y ago

I think it depends on how large your codebase is and how many dependencies it has. The first issue is that any new 3.x release is going to have bugs introduced by new features, and possibly regressions in old ones. This could cause your current code to behave differently, as well as cause issues if someone immediately uses a new feature. The other issue is the more dependencies you have, the more likely one of them will not work with the newest version. Usually these things get sorted out pretty quickly, but it's frustrating to get a bunch of errors. I typically wait until at least the first bug fix release (3.11.1 for example) to try out a new Python version. These often have most of the common issues fixed, and give enough time for all the popular dependencies to catch up. This is from my experience maintaining a million+ line codebase that started at Python 2.6 and was updated to 3.8 as new versions came out.

reuven 4y ago

There are very, very few breaking changes within Python 3. If you're an individual user, then upgrading to the latest version is almost certainly a good idea. But there are lots of companies that release software using Python. To ensure a stable production environment, they upgrade more slowly, or on a schedule that's distinct from the Python release cycle. Also, companies often don't just let people upgrade things themselves. The IT department schedules a company-wide Python upgrade, for example, testing the effects beforehand. I have several training clients still using 3.7 and 3.8, not because things would break with 3.10, but because they haven't fully vetted the newest version, and they're being super conservative. I do wonder if companies will be a bit faster to upgrade over the coming years, given that execution speed is expected to improve rather dramatically. Upgrading won't be a matter of some nice-to-have features, but rather of real savings in system usage.

smitty1e 4y ago
> There are very, very few breaking changes within Python 3. But will structural pattern matching be back-ported, though? Certainly an incompatibility, if not a breaking change per se.
- kdmccormick 4y ago
  GP means there are very few backwards-incompatible changes in new releases. There are plenty of forwards-incompatible ones, though.
- fnord123 4y ago
  How is pattern matching a breaking change? If your code doesn't have it how would it break?
  - nedbat 4y ago
    Some people might mistakenly assume that "match" and "case" are now hard keywords, which would of course break lots of code. But they are soft keywords, so there's no issue.

david2ndaccount 4y ago

There are breaking changes in the C api occasionally, so that can be a good reason to not upgrade. Sometimes an extension needs to use the unstable API.

theamk 4y ago

If using python for large project, sure, use latest version. If writing small tools, consder limiting yourself to system python and its stdlib / system-packaged libraries. A script which ypu can download as a single file, immediately run, and immediately make some changes in, all without worrying about venvs or other setup, is a beautiful thing. This works especially well if you are writing internal tools and your company has standarts like "Everyone is on Ubuntu 20.04 or 22.04"

heavyset_go 4y ago

Sometimes. If you're relying on a toolchain that's parsing and analyzing Python code, things like compilers and mypy will sometimes lag behind new syntax, bytecode and C API changes. Until recently mypy didn't have full support for match statements, and its corresponding compiler, mypyc, still doesn't. Same goes, or went, for full support for match statements in other compilers, as well.

vorhemus 4y ago

Some popular packages are Python version dependent. For example, I was using Python 3.10 early and no Python wheels for 3.10 were released for PyTorch back then. The only thing you can do in this case is to either wair for the publisher to provide these packages or use a lower version Python interpreter.

itsgrimetime 4y ago
The 3.10 interpreter would still run 3.9-compatible Python code though, right?
- faho 4y ago
  For python code, yeah, most of it. Python does sometimes have backwards-incompatible changes, e.g. for 3.10 they removed a bunch of stdlib modules and methods, like the "formatter" and "parser" module [0]. So if you used those, your code wouldn't work in 3.10. But the main reason to wait for wheels (which is pythonese for "pre-built packages") is if they use native code (like C or rust) and you would have to compile them yourself otherwise, which increases installation time quite a bit. (this was also the reason why Alpine was a bad choice for python containers for a long time because it uses musl and there were only wheels for glibc available. AFAIK musl wheels exist now so that isn't relevant anymore) [0]: https://docs.python.org/3/whatsnew/3.10.html#removed https://docs.python.org/3/whatsnew/3.10.html#removed
  - diarrhea 4y ago
    I always wondered what’s keeping packages back from having wheels ready the day a release drops, or way before even? I imagine often it’s just another parameter in a CI somewhere, where things mostly “just work” because backward-incompatible breakages have become much rarer.
    - singhrac 4y ago
      It's usually a question of having the resources (usually developers to debug issues). IIRC pretty much all Windows wheels are maintained by one guy, Christoph Gohlke, he was a godsend when I worked on a Windows laptop. I owe that guy many beers.

0x008 4y ago

Yea, if you use standard docker images o you want to use the python version they come with which is usually 3.8 (Ubuntu 20.04) or 3.6 (Ubuntu 18.04)

arjvik 4y ago
Why not use the library/python image with the specific version over library/Ubuntu?
- 0x008 4y ago
  I don’t understand your question, but to illustrate the point: We work often with 3rd party libraries and images provided by those third parties. These are often based straight up on Ubuntu. Nvidia CUDA image is an example of this. Then you are stuck with their python version. And it’s not as easy as installing your own python version into the image because python dependencies inside those 3rd party images might be installed using OS package managers and not pip so then they are installed to the wrong python version etc. It’s a rabbit hole. The problem is that at some point Debian/ubuntu decided to repackage python dependencies as apt packages.

OJFord 4y ago

There have been breaking changes, I don't think anything too difficult to overcome, but it's easy to pin it intending to fix later, and fall behind. (e.g. 3.9-3.10 something to do with dataclasses iirc)

lynndotpy 4y ago

I find Conda packages often have a maximum version that's a bit out of date, sometimes several versions behind. I'm sure using 3.10 instead of 3.9 would be fine for most packages, but I don't know if it can be "forced"

darkarmani 4y ago
Compiled packages have ABI restrictions. Even conda-forge puts major version constraints on their packages at build time (ie: built on 3.9: >=3.9,<3.10.0a).

ramraj07 4y ago

Unfortunately some python packages are not reliably able to move across versions. I’ve had issues with sqlalchemy and celery before. Let’s not get started with the python packages from snowflake.

dragonwriter 4y ago

If you are dependent on binary packages rather than building extensions from source (more common on Windows), there may not be prebuilt packages for some libraries for the newest version.

Steltek 4y ago

If you're writing internal tools for your teammates, don't use the latest release. Your coworkers will almost certainly be on something older and you'll get super weird bugs. And you'll get the most bizarre sounding pushback when you tell them to install a newer version of Python.

digisign 4y ago

There are many little incompatibilities over 3.x. Especially at 3.10/11 where deprecations are finally being removed. Few big changes, as from 2.x. Wheels typically lag however. Release[-2] or Release[-3] are good choices. Definitely upgrade from [-4] or lower.

InfoSecErik 4y ago

For my Python things, I aimed to be compatible with the oldest supported Python for people on older OS's or who aren't constantly updating their Python installations. That way my stuff was still accessible to them.