7 ms·

There was a discussion the other day about how Python devs apparently don't care enough for backwards compatibility. I pointed out that I've often gotten Python

by andai 2y ago

There was a discussion the other day about how Python devs apparently don't care enough for backwards compatibility. I pointed out that I've often gotten Python 2 code running on Python 3 by just changing print to print(). But then a few hours later, I tried running a very small project I wrote last year and it turned out that a bunch of my dependencies had changed their APIs. I've had similar (and much worse) experiences trying to get older code with dependencies running. My meaning with this comment is, that if the average developer's reality is that backwards compatibility isn't really a thing anyway, then we are already paying for that downside so we might as well get some upside there, is my reasoning.

almostgotcaught 2y ago
> I pointed out that I've often gotten Python 2 code running on Python 3 by just changing print to print(). ... > I wrote last year and it turned out that a bunch of my dependencies had changed their APIs these two things have absolutely nothing to do with each other - couldn't be a more apples to oranges comparison if you tried
- andai 2y ago
  I ran into both of these things in the same context, which is "the difficulty involved in getting old code working on the latest Python environment", which I understood as the context of this discussion.
klysm 2y ago
So pin your deps? Language backwards compatibility and an API from some random package changing are completely distinct.
- epistasis 2y ago
  Pinning deps is discouraged by years of Python practice. And going back to a an old project and finding versions that work, a year or more later, might be nigh on impossible. Last week I was trying to install snakemake via Conda, and couldn't find any way to satisfy dependencies at all, so it's not just pypi, and pip tends to be one of the more forgiving version dependency managers. It's not just Python, trying to get npm to load the requirements has stopped me from compiling about half of the projects I've tried to build (which is not a ton of projects). And CRAN in the R universe can have similar problems as projects age.
  - klysm 2y ago
    That doesn’t match my experience at all. I have many Python projects going back years that all work fine with pinned dependencies
  - vosper 2y ago
    > Pinning deps is discouraged by years of Python practice. I'm not sure it is discouraged so much as just not what people did in Python-land for a long time. It's obviously the right thing to do, it's totally doable, it's just inertia and habit that might mean it isn't done.
    - lmm 2y ago
      > I'm not sure it is discouraged so much as just not what people did in Python-land for a long time. It's obviously the right thing to do, it's totally doable, it's just inertia and habit that might mean it isn't done. Pinning obviously the wrong thing, it only works if everyone does it and if everyone does it then making changes becomes very hard. The right thing is to have deterministic dependency resolution so that dependencies don't change under you.
      - WillDaSilva 2y ago
        When they suggest you pin your dependencies, they don't just mean your direct dependencies, but rather all transitive dependencies. You can take this further by having a lock file that account for different Python versions, operating systems, and CPI architectures – for instance , by using UV or Poetry – but a simple `pip freeze` is often sufficient.
        epistasis 2y ago
        That works for your project, but then nobody can include you as a library without conflicts. But having that lock file will allow somebody to reconstruct your particular moment in time in the future. Its just that those lock files do not exist for 99.9% of Python projects in time.
- nomel 2y ago
  > So pin your deps? Which is, fairly often, pinning your python version.
- BerislavLopac 2y ago
  Yes and no. There are different types of dependencies, and there are different rules for them, but here's an overview of the best practices: 1. For applications, scripts and services (i.e. "executable code"), during development, pin your direct dependencies; ideally to the current major or minor version, depending how much you trust their their authors to follow SemVer. Also make sure you regularly update and retest the versions to make sure you don't miss any critical updates. You should not explicitly pin your transitive dependencies, i.e. the dependencies of your direct dependencies -- at least unless you know specifically that certain versions will break your app (and even then it is better to provide a range than a single version). 2. For production builds of the above, lock each of your dependencies (including the transitive ones) to specific version and source. It is not really viable to do it by hand, but most packaging tools -- pip, Poetry, PDM, uv... -- will happily do that automatically for you. Unfortunately, Python still doesn't have a standard lock format, so most tools provide their own lock file; the closest thing to a standard we have at the moment is pip's requirements file [0]. Besides pinned versions, a lock file will also include the source where the packages are to be retrieved from (pip's requirements file may omit it, but it's then implicitly assumed to be PyPI); it can (and should, really) also provide hashes for the given packages, strengthening the confidence that you're downloading the correct packages. 3. Finally, when developing libraries (i.e. "redistributable code"), you should never pin your dependencies at all -- or, at most, you can specify the minimum versions that you know that work and have tested against. That is because you have no control over the environment the code will eventually be used and executed in, and arbitrary limitations like that might (and often will) prevent your users to update some other crucial dependency. Of course, the above does not apply if you know that a certain version range will break your code. It that case you should most definitely exclude it from your specification -- but you should also update your code as soon as possible. Libraries should also clearly specify which versions of Python they support, and should be regularly tested against each of those versions; it is also recommended that the minimal supported version is regularly reviewed and increased as new versions of Python get released [1]. For more clarity on abstract vs concrete dependencies, I recommend the great article by Donald Stufft from 2013 [2]; and for understanding why top-binding (i.e. limiting the top version pin) should be avoided there is a lengthy but very detailed analysis by Henry Schreiner [3]. [0] https://pip.pypa.io/en/stable/reference/requirements-file-format/ https://pip.pypa.io/en/stable/reference/requirements-file-fo... [1] https://devguide.python.org/versions/ https://devguide.python.org/versions/ [2] https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/ [3] https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/
adamc 2y ago
It's hard to comment on this without knowing more about the dependencies and when/how they changed their APIs. I would say if it was a major version change, that isn't too shocking. For a minor version change, it should be. Stuff that is actually included with Python tends to be more stable than random Pypi packages, though. NPM packages also sometimes change. That's the world.
- Timon3 2y ago
  The big difference is that npm will automatically (since 2017) save a version range to the project metadata, and will automatically create this metadata file if it doesn't exist. Same for other package managers in the Node world. I just installed Python 3.13 with pip 24.2, created a venv and installed a package - and nothing, no file was created and nothing was saved. Even if I touch requirements.txt and pyproject.toml, pip doesn't save anything about the package. This creates a massive gap in usability of projects by people not very familiar with the languages. Node-based projects sometimes have issues because dependencies changed without respecting semver, but Python projects often can't be installed and you have no idea why without spending lots of time looking through versions. Of course there are other package managers for Python that do this better, but pip is still the de-facto default and is often used in tutorials for new developers. Hopefully uv can improve things!
  - elashri 2y ago
    > Of course there are other package managers for Python that do this better I think if you are comparing with what NPM does then you would have to say that native pip can do that too. It is just one command `pip freeze > requirements.txt` It does include everything in the venv (or in you environment in general) but if you stick to only add required things (one venv for each project) then you will get requirements.txt files
    - whyever 2y ago
      I don't think this is the same. Does it also cover transitive dependencies?
saurik 2y ago
The Python 2 to 3 thing was worse when they started: people who made the mistake of falling for the rhetoric to port to python3 early on had a much more difficult time as basic things like u"" were broken under an argument that they weren't needed anymore; over time the porting process got better as they acquiesced and unified the two languages a bit. I thereby kind of feel like this might have happened in the other direction: a ton of developers seem to have become demoralized by python3 and threw up their hands in defeat of "backwards compatibility isn't going to happen anyway", and now we live in a world with frozen dependencies running in virtual environments tied to specific copies of Python.
- Shawnj2 2y ago
  Honestly I think a big issue is that it’s not just legacy code, it’s also legacy code which depends on old dependency versions. Eg there’s an internal app where I work stuck on 3.8.X because it uses deprecated pandas syntax and is too complicated to rewrite for a newer version easily.
dataflow 2y ago
> Python 2 code running on Python 3 by just changing print to print(). This was very much the opposite of my experience. Consider yourself lucky.
- dietr1ch 2y ago
  This migration took the industry years because it was not that simple.
  - selcuka 2y ago
    > This migration took the industry years because it was not that simple. It was not that simple, but it was not that hard either. It took the industry years because Python 2.7 was still good enough, and the tangible benefits of migrating to Python 3 didn't justify the effort for most projects. Also some dependencies such as MySQL-python never updated to Python 3, which was also an issue for projects with many dependencies.
- LtWorf 2y ago
  Maybe his application was an hello world!
salomonk_mur 2y ago
What APIs were broken? They couldn't be in the standard library. If the dependency was in external modules and you didn't have pinned versions, then it is to be expected (in almost any active language) that some APIs will break.
- dagw 2y ago
  They couldn't be in the standard library. Why not? Python does make breaking changes to the standard library when going from 3.X to 3.X+1 quite regularly.
  - noitpmeder 2y ago
    Only usually after YEARS of deprecation warnings
    - physicsguy 2y ago
      Async was a good example where that didn’t happen, but to be fair to the maintainers, it was fairly experimental
- LtWorf 2y ago
  Python drops modules from the standard library all the time these days. It's a pain in the ass. Now even asyncore is gone -_-' Have fun rewriting all the older async applications!
kwertzzz 2y ago
Sadly, several python projects do not use semantic versioning, for example xarray [0] and dask. Numpy can make backward incompatible changes after a warning for two releases[1]. In general, the python packaging docs do not really read as an endorsement of semantic versioning [2]: > A majority of Python projects use a scheme that resembles semantic versioning. However, most projects, especially larger ones, do not strictly adhere to semantic versioning, since many changes are technically breaking changes but affect only a small fraction of users... [0] https://github.com/pydata/xarray/issues/6176 https://github.com/pydata/xarray/issues/6176 [1] https://numpy.org/doc/stable/dev/depending_on_numpy.html https://numpy.org/doc/stable/dev/depending_on_numpy.html [2] https://packaging.python.org/en/latest/discussions/versioning/ https://packaging.python.org/en/latest/discussions/versionin...
LtWorf 2y ago
I'd drop libraries that do like that.
musicale 2y ago
Even after it finished burning a billion lines of python 2 code (largely unnecessarily imho) python 3 seems to retain an unhealthy contempt for backward compatibility. I have had similar experiences where python 3 projects require a particular version of python 3 in order to run. I like python (and swift for that matter) but I don't like the feeling that I am building on quicksand. Java, C++, and vanilla javascript seem more durable.

by andai 2y ago

almostgotcaught 2y ago

> I pointed out that I've often gotten Python 2 code running on Python 3 by just changing print to print(). ... > I wrote last year and it turned out that a bunch of my dependencies had changed their APIs these two things have absolutely nothing to do with each other - couldn't be a more apples to oranges comparison if you tried

andai 2y ago
I ran into both of these things in the same context, which is "the difficulty involved in getting old code working on the latest Python environment", which I understood as the context of this discussion.

klysm 2y ago

So pin your deps? Language backwards compatibility and an API from some random package changing are completely distinct.

epistasis 2y ago
Pinning deps is discouraged by years of Python practice. And going back to a an old project and finding versions that work, a year or more later, might be nigh on impossible. Last week I was trying to install snakemake via Conda, and couldn't find any way to satisfy dependencies at all, so it's not just pypi, and pip tends to be one of the more forgiving version dependency managers. It's not just Python, trying to get npm to load the requirements has stopped me from compiling about half of the projects I've tried to build (which is not a ton of projects). And CRAN in the R universe can have similar problems as projects age.
- klysm 2y ago
  That doesn’t match my experience at all. I have many Python projects going back years that all work fine with pinned dependencies
- vosper 2y ago
  > Pinning deps is discouraged by years of Python practice. I'm not sure it is discouraged so much as just not what people did in Python-land for a long time. It's obviously the right thing to do, it's totally doable, it's just inertia and habit that might mean it isn't done.
  - lmm 2y ago
    > I'm not sure it is discouraged so much as just not what people did in Python-land for a long time. It's obviously the right thing to do, it's totally doable, it's just inertia and habit that might mean it isn't done. Pinning obviously the wrong thing, it only works if everyone does it and if everyone does it then making changes becomes very hard. The right thing is to have deterministic dependency resolution so that dependencies don't change under you.
    - WillDaSilva 2y ago
      When they suggest you pin your dependencies, they don't just mean your direct dependencies, but rather all transitive dependencies. You can take this further by having a lock file that account for different Python versions, operating systems, and CPI architectures – for instance , by using UV or Poetry – but a simple `pip freeze` is often sufficient.
      - epistasis 2y ago
        That works for your project, but then nobody can include you as a library without conflicts. But having that lock file will allow somebody to reconstruct your particular moment in time in the future. Its just that those lock files do not exist for 99.9% of Python projects in time.
nomel 2y ago
> So pin your deps? Which is, fairly often, pinning your python version.
BerislavLopac 2y ago
Yes and no. There are different types of dependencies, and there are different rules for them, but here's an overview of the best practices: 1. For applications, scripts and services (i.e. "executable code"), during development, pin your direct dependencies; ideally to the current major or minor version, depending how much you trust their their authors to follow SemVer. Also make sure you regularly update and retest the versions to make sure you don't miss any critical updates. You should not explicitly pin your transitive dependencies, i.e. the dependencies of your direct dependencies -- at least unless you know specifically that certain versions will break your app (and even then it is better to provide a range than a single version). 2. For production builds of the above, lock each of your dependencies (including the transitive ones) to specific version and source. It is not really viable to do it by hand, but most packaging tools -- pip, Poetry, PDM, uv... -- will happily do that automatically for you. Unfortunately, Python still doesn't have a standard lock format, so most tools provide their own lock file; the closest thing to a standard we have at the moment is pip's requirements file [0]. Besides pinned versions, a lock file will also include the source where the packages are to be retrieved from (pip's requirements file may omit it, but it's then implicitly assumed to be PyPI); it can (and should, really) also provide hashes for the given packages, strengthening the confidence that you're downloading the correct packages. 3. Finally, when developing libraries (i.e. "redistributable code"), you should never pin your dependencies at all -- or, at most, you can specify the minimum versions that you know that work and have tested against. That is because you have no control over the environment the code will eventually be used and executed in, and arbitrary limitations like that might (and often will) prevent your users to update some other crucial dependency. Of course, the above does not apply if you know that a certain version range will break your code. It that case you should most definitely exclude it from your specification -- but you should also update your code as soon as possible. Libraries should also clearly specify which versions of Python they support, and should be regularly tested against each of those versions; it is also recommended that the minimal supported version is regularly reviewed and increased as new versions of Python get released [1]. For more clarity on abstract vs concrete dependencies, I recommend the great article by Donald Stufft from 2013 [2]; and for understanding why top-binding (i.e. limiting the top version pin) should be avoided there is a lengthy but very detailed analysis by Henry Schreiner [3]. [0] https://pip.pypa.io/en/stable/reference/requirements-file-format/ https://pip.pypa.io/en/stable/reference/requirements-file-fo... [1] https://devguide.python.org/versions/ https://devguide.python.org/versions/ [2] https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/ [3] https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/

adamc 2y ago

It's hard to comment on this without knowing more about the dependencies and when/how they changed their APIs. I would say if it was a major version change, that isn't too shocking. For a minor version change, it should be. Stuff that is actually included with Python tends to be more stable than random Pypi packages, though. NPM packages also sometimes change. That's the world.

Timon3 2y ago
The big difference is that npm will automatically (since 2017) save a version range to the project metadata, and will automatically create this metadata file if it doesn't exist. Same for other package managers in the Node world. I just installed Python 3.13 with pip 24.2, created a venv and installed a package - and nothing, no file was created and nothing was saved. Even if I touch requirements.txt and pyproject.toml, pip doesn't save anything about the package. This creates a massive gap in usability of projects by people not very familiar with the languages. Node-based projects sometimes have issues because dependencies changed without respecting semver, but Python projects often can't be installed and you have no idea why without spending lots of time looking through versions. Of course there are other package managers for Python that do this better, but pip is still the de-facto default and is often used in tutorials for new developers. Hopefully uv can improve things!
- elashri 2y ago
  > Of course there are other package managers for Python that do this better I think if you are comparing with what NPM does then you would have to say that native pip can do that too. It is just one command `pip freeze > requirements.txt` It does include everything in the venv (or in you environment in general) but if you stick to only add required things (one venv for each project) then you will get requirements.txt files
  - whyever 2y ago
    I don't think this is the same. Does it also cover transitive dependencies?

saurik 2y ago

The Python 2 to 3 thing was worse when they started: people who made the mistake of falling for the rhetoric to port to python3 early on had a much more difficult time as basic things like u"" were broken under an argument that they weren't needed anymore; over time the porting process got better as they acquiesced and unified the two languages a bit. I thereby kind of feel like this might have happened in the other direction: a ton of developers seem to have become demoralized by python3 and threw up their hands in defeat of "backwards compatibility isn't going to happen anyway", and now we live in a world with frozen dependencies running in virtual environments tied to specific copies of Python.

Shawnj2 2y ago
Honestly I think a big issue is that it’s not just legacy code, it’s also legacy code which depends on old dependency versions. Eg there’s an internal app where I work stuck on 3.8.X because it uses deprecated pandas syntax and is too complicated to rewrite for a newer version easily.

dataflow 2y ago

> Python 2 code running on Python 3 by just changing print to print(). This was very much the opposite of my experience. Consider yourself lucky.

dietr1ch 2y ago
This migration took the industry years because it was not that simple.
- selcuka 2y ago
  > This migration took the industry years because it was not that simple. It was not that simple, but it was not that hard either. It took the industry years because Python 2.7 was still good enough, and the tangible benefits of migrating to Python 3 didn't justify the effort for most projects. Also some dependencies such as MySQL-python never updated to Python 3, which was also an issue for projects with many dependencies.
LtWorf 2y ago
Maybe his application was an hello world!

salomonk_mur 2y ago

What APIs were broken? They couldn't be in the standard library. If the dependency was in external modules and you didn't have pinned versions, then it is to be expected (in almost any active language) that some APIs will break.

dagw 2y ago
They couldn't be in the standard library. Why not? Python does make breaking changes to the standard library when going from 3.X to 3.X+1 quite regularly.
- noitpmeder 2y ago
  Only usually after YEARS of deprecation warnings
  - physicsguy 2y ago
    Async was a good example where that didn’t happen, but to be fair to the maintainers, it was fairly experimental
LtWorf 2y ago
Python drops modules from the standard library all the time these days. It's a pain in the ass. Now even asyncore is gone -_-' Have fun rewriting all the older async applications!

kwertzzz 2y ago

Sadly, several python projects do not use semantic versioning, for example xarray [0] and dask. Numpy can make backward incompatible changes after a warning for two releases[1]. In general, the python packaging docs do not really read as an endorsement of semantic versioning [2]: > A majority of Python projects use a scheme that resembles semantic versioning. However, most projects, especially larger ones, do not strictly adhere to semantic versioning, since many changes are technically breaking changes but affect only a small fraction of users... [0] https://github.com/pydata/xarray/issues/6176 https://github.com/pydata/xarray/issues/6176 [1] https://numpy.org/doc/stable/dev/depending_on_numpy.html https://numpy.org/doc/stable/dev/depending_on_numpy.html [2] https://packaging.python.org/en/latest/discussions/versioning/ https://packaging.python.org/en/latest/discussions/versionin...

LtWorf 2y ago

I'd drop libraries that do like that.

musicale 2y ago

Even after it finished burning a billion lines of python 2 code (largely unnecessarily imho) python 3 seems to retain an unhealthy contempt for backward compatibility. I have had similar experiences where python 3 projects require a particular version of python 3 in order to run. I like python (and swift for that matter) but I don't like the feeling that I am building on quicksand. Java, C++, and vanilla javascript seem more durable.