4 ms·

If you publish a Python library with pinned dependencies, your code is broken as soon as someone tries to use it with another Python library with pinned depende

If you publish a Python library with pinned dependencies, your code is broken as soon as someone tries to use it with another Python library with pinned dependencies, unless you happened to pin exactly the same version of the dependencies you have in common. Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. There are tools like Pipenv and Poetry to make that easy. This is less of an issue in (say) Node.js, where you can have multiple different versions of a library installed in different branches of the dependency tree. (Though Node.js also has a strong semver culture that almost always works well enough that pinning exact versions isn’t necessary.)

acidbaseextract 6y ago
The most frustrating thing is that pip doesn't make it easy to use more loose declared dependencies while freezing to actual concrete dependencies for deployment. Everybody rolls their own. > Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. Is the pypi package awscli an application or a library? poetry is frustrating in that it doesn't allow you to override a library's declared requirements to break conflicts. They refuse to add support [1][2] for the feature too. awscli for example causes huge package conflict issues that make poetry unusable. It's almost impossible not to run into a requirement conflict with awscli if you're using a broad set of packages, even though awscli will operate happily with a more broad set of requirements than it declares. [1] https://github.com/python-poetry/poetry/issues/697 https://github.com/python-poetry/poetry/issues/697 [2] https://github.com/python-poetry/poetry/issues/697#issuecomment-744235786 https://github.com/python-poetry/poetry/issues/697#issuecomm...
- anderskaseorg 6y ago
  For this purpose, I’m defining a “library” as any PyPI package that you expect to be able to install alongside other PyPI packages. This includes some counterintuitive ones like mypy, which needs to extract types from packages in the same environment as the code it’s checking. The awscli documentation recommends installing it into its own virtualenv, in which case pinned dependencies may be reasonable. There are tools like pipx to automate that. Though in practice, there are reasons that installing applications into their own virtualenv might be inconvenient, inefficient, or impossible. And even when it’s possible, it still comes with the risk of missing security updates unless upstream is doing a really good job of staying on top of them. I don’t think that respecting declared dependency bounds is a Poetry bug. Pip respects them too (at least as of 20.3, which enables the new resolver by default: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020 https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip...). If a package declares unhelpful bounds, the package should be fixed. (And yes, that means its maintainer might have to deal with some extra issues being filed—that’s part of the job.)
- sagichmal 6y ago
  > Is the pypi package awscli an application or a library? Hopefully a library! As hopefully the AWS command-line interface is maintained and distributed separately from any SDK that powers it...
  - X-Istence 6y ago
    boto core/boto/boto 3 are the libraries, which awscli drives.
- orf 6y ago
  Why on earth would you ever add awscli as a dependency? That makes very little sense. It’s an application (that is no longer distributed via pypi). You should use boto3
u801e 6y ago
> Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. This is essentially what we do where I work. When we maked a tagged release, we will create a new virtual environment, run a pip install, run all the tests and then run pip freeze. The output of pip freeze is what we use for the install_requires parameter in the setup method in setup.py. That said, a library could certainly could update their old releases with a patch release and specify a <= requirement on a particular dependency when versions newer than that no longer work. That said, it would be a bit of work since indirect dependencies would also have to be accounted for as well.

acidbaseextract 6y ago

The most frustrating thing is that pip doesn't make it easy to use more loose declared dependencies while freezing to actual concrete dependencies for deployment. Everybody rolls their own. > Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. Is the pypi package awscli an application or a library? poetry is frustrating in that it doesn't allow you to override a library's declared requirements to break conflicts. They refuse to add support [1][2] for the feature too. awscli for example causes huge package conflict issues that make poetry unusable. It's almost impossible not to run into a requirement conflict with awscli if you're using a broad set of packages, even though awscli will operate happily with a more broad set of requirements than it declares. [1] https://github.com/python-poetry/poetry/issues/697 https://github.com/python-poetry/poetry/issues/697 [2] https://github.com/python-poetry/poetry/issues/697#issuecomment-744235786 https://github.com/python-poetry/poetry/issues/697#issuecomm...

anderskaseorg 6y ago
For this purpose, I’m defining a “library” as any PyPI package that you expect to be able to install alongside other PyPI packages. This includes some counterintuitive ones like mypy, which needs to extract types from packages in the same environment as the code it’s checking. The awscli documentation recommends installing it into its own virtualenv, in which case pinned dependencies may be reasonable. There are tools like pipx to automate that. Though in practice, there are reasons that installing applications into their own virtualenv might be inconvenient, inefficient, or impossible. And even when it’s possible, it still comes with the risk of missing security updates unless upstream is doing a really good job of staying on top of them. I don’t think that respecting declared dependency bounds is a Poetry bug. Pip respects them too (at least as of 20.3, which enables the new resolver by default: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020 https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip...). If a package declares unhelpful bounds, the package should be fixed. (And yes, that means its maintainer might have to deal with some extra issues being filed—that’s part of the job.)
sagichmal 6y ago
> Is the pypi package awscli an application or a library? Hopefully a library! As hopefully the AWS command-line interface is maintained and distributed separately from any SDK that powers it...
- X-Istence 6y ago
  boto core/boto/boto 3 are the libraries, which awscli drives.
orf 6y ago
Why on earth would you ever add awscli as a dependency? That makes very little sense. It’s an application (that is no longer distributed via pypi). You should use boto3

u801e 6y ago

> Python libraries should not pin dependencies. _Applications_ can pin dependencies, including all recursive dependencies of their libraries. This is essentially what we do where I work. When we maked a tagged release, we will create a new virtual environment, run a pip install, run all the tests and then run pip freeze. The output of pip freeze is what we use for the install_requires parameter in the setup method in setup.py. That said, a library could certainly could update their old releases with a patch release and specify a <= requirement on a particular dependency when versions newer than that no longer work. That said, it would be a bit of work since indirect dependencies would also have to be accounted for as well.