4 ms·

> I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python pa

> I haven't found an instance where python3-X and python3-Y can't be installed because one requires python3-Z v1.2 and the other python3-Z v2.1. With Python packages, that happens way more often. Exactly, because the packages were created/updated at a point in time were they were compatible: a distro ships python 3.x with packages of python modules that are compatible with python 3.x. Now if you as a user want to use a python package that works with python 3.y, you either have to wait, or install python 3.y, or use a container of python 3.y. Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work. > Also, distro package managers keep track of everything installed, while python ones (at least pip I'm 100% sure) don't. Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;) > You could install a package which upgrades another one, and that breaks another that you had installed before and pip won't say a word. Let's agree to disagree right there: - in your point of view: this is a problem in pip - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning

gjulianm 5y ago
> Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work. That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation. > Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;) I know, but pip doesn't track all of those all the time. An example that has happened to me multiple times: install package X that depends on Y <= 1.0. Good, pip installs the proper version. Now, on another command, install package Z that depends on Y >= 2.0. Pip will install Y >= 2.0 and won't care that package X is now broken. > - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning Regardless on your views on pinning, it's a problem in pip. A version conflict should be reported as an installation failure, not allow you to continue. And again, version restrictions will always be there. The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python. Even without version pinning, you'll still have packages breaking because another one was updated. Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs.
- 1337shadow 5y ago
  > That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation. You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever? > A version conflict should be reported as an installation failure, not allow you to continue. Please don't make this mandatory. > Pip will install Y >= 2.0 and won't care that package X is now broken. Fine, I'll just quickly fix X and open a pull request, like I probably did a hundred times, then eventually if necessary deploy my fork with the fix meanwhile they do their maintenance release. Should we not take the responsibility of the dependencies we use and contribute back?? > The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python. I don't understand why, but I'm talking about "live at latest release", not "at master". > Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs. Then the package can just paste the code of the feature in their own, until a new lib does it, I remember having to do that twice in 20 years (except I didn't just "paste" it, but implemented a much smaller version). Overall, it seems my approach produces versions of my software and software that I depend on that are compatible with all versions because you can always use an earlier version if you really want to deploy on an old python or whatnot, whereas you approach leads to broken packages, tech debt, and blaming the package manager.
  - gjulianm 5y ago
    > You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever? No, it means that you can't be fully sure that upgrading everything to the latest version is not going to break anything. And right now, you can't. You can only do that in controlled environments (say, a distro's official package repositories), not with Python packages. > Please don't make this mandatory. It is mandatory in quite a lot of package managers. APT, for example, will refuse to install packages with conflicting versions or breaking things. It's better to fail early with a clear message than to have a later failure where the cause is unclear. > Fine, I'll just quickly fix X and open a pull request, like I probably did a hundred times, then eventually if necessary deploy my fork with the fix meanwhile they do their maintenance release. That's optimistic. What about detecting which package is causing the issue? What if the fix is not quick? What if the package developers are working on compatibility but it's going to take time? > I don't understand why, but I'm talking about "live at latest release", not "at master". Problem is similar. You can't live at latest release with software coming from wildly different developers, with wildly different policies on compatibility, versioning, breaking changes, bugs... > Then the package can just paste the code of the feature in their own, until a new lib does it, I remember having to do that twice in 20 years (except I didn't just "paste" it, but implemented a much smaller version). Again, pretty optimistic. You won't be able to do that with all packages. For example, right now I have a project that needs to work on Python 3.6 (among other packages). Latest numpy versions dropped support for Python 3.6, so the project needs to restrict numpy versions to maintain compatibility. I can't just take the latest numpy and patch it for Python 3.6. > Overall, it seems my approach produces versions of my software and software that I depend on that are compatible with all versions because you can always use an earlier version if you really want to deploy on an old python or whatnot, whereas you approach leads to broken packages, tech debt, and blaming the package manager. You also spend time doing maintenance and debugging on new installations to debug and fix compatibility issues, when those upgrades might not bring any value to the product. In my case, I do that debugging and fixing in a controlled environment, only when I decide to upgrade the packages, and maybe revert/pin the ones that don't have quick fixes. That's the difference. Once a given package is released/packaged for distribution, I want dependencies to be fixed so that every new installation works, and doesn't fail if some developer decided to break things that day.

gjulianm 5y ago

> Again, if all python package were up to date as in "working together at this point in time" then `pip upgrade` would work. That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation. > Are you sure you checked in the `*.dist-info` directories? There should be one per package, containing: a METADATA file, with all dependency versions, a RECORD file, with every installed file and their hash, and much more! see for yourself ;) I know, but pip doesn't track all of those all the time. An example that has happened to me multiple times: install package X that depends on Y <= 1.0. Good, pip installs the proper version. Now, on another command, install package Z that depends on Y >= 2.0. Pip will install Y >= 2.0 and won't care that package X is now broken. > - in my point of view: this is a maintenance problem in the python packages themselves, caused by the general practice of pinning Regardless on your views on pinning, it's a problem in pip. A version conflict should be reported as an installation failure, not allow you to continue. And again, version restrictions will always be there. The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python. Even without version pinning, you'll still have packages breaking because another one was updated. Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs.

1337shadow 5y ago
> That is impossible, it will never happen. Packages will have different maintenance cycles, some will get deprecated, others abandoned... You can't base your upgrade policy on an impossible situation. You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever? > A version conflict should be reported as an installation failure, not allow you to continue. Please don't make this mandatory. > Pip will install Y >= 2.0 and won't care that package X is now broken. Fine, I'll just quickly fix X and open a pull request, like I probably did a hundred times, then eventually if necessary deploy my fork with the fix meanwhile they do their maintenance release. Should we not take the responsibility of the dependencies we use and contribute back?? > The "live at master" philosophy only works for small groups of similar output capacity. It won't work for an ecosystem as wide as Python. I don't understand why, but I'm talking about "live at latest release", not "at master". > Sometimes it will be necessary, such as for example a package dropping support for a feature that another one needs. Then the package can just paste the code of the feature in their own, until a new lib does it, I remember having to do that twice in 20 years (except I didn't just "paste" it, but implemented a much smaller version). Overall, it seems my approach produces versions of my software and software that I depend on that are compatible with all versions because you can always use an earlier version if you really want to deploy on an old python or whatnot, whereas you approach leads to broken packages, tech debt, and blaming the package manager.
- gjulianm 5y ago
  > You realize that if that was "impossible" and "never happening", then absolutely no Python environment would be working ever? No, it means that you can't be fully sure that upgrading everything to the latest version is not going to break anything. And right now, you can't. You can only do that in controlled environments (say, a distro's official package repositories), not with Python packages. > Please don't make this mandatory. It is mandatory in quite a lot of package managers. APT, for example, will refuse to install packages with conflicting versions or breaking things. It's better to fail early with a clear message than to have a later failure where the cause is unclear. > Fine, I'll just quickly fix X and open a pull request, like I probably did a hundred times, then eventually if necessary deploy my fork with the fix meanwhile they do their maintenance release. That's optimistic. What about detecting which package is causing the issue? What if the fix is not quick? What if the package developers are working on compatibility but it's going to take time? > I don't understand why, but I'm talking about "live at latest release", not "at master". Problem is similar. You can't live at latest release with software coming from wildly different developers, with wildly different policies on compatibility, versioning, breaking changes, bugs... > Then the package can just paste the code of the feature in their own, until a new lib does it, I remember having to do that twice in 20 years (except I didn't just "paste" it, but implemented a much smaller version). Again, pretty optimistic. You won't be able to do that with all packages. For example, right now I have a project that needs to work on Python 3.6 (among other packages). Latest numpy versions dropped support for Python 3.6, so the project needs to restrict numpy versions to maintain compatibility. I can't just take the latest numpy and patch it for Python 3.6. > Overall, it seems my approach produces versions of my software and software that I depend on that are compatible with all versions because you can always use an earlier version if you really want to deploy on an old python or whatnot, whereas you approach leads to broken packages, tech debt, and blaming the package manager. You also spend time doing maintenance and debugging on new installations to debug and fix compatibility issues, when those upgrades might not bring any value to the product. In my case, I do that debugging and fixing in a controlled environment, only when I decide to upgrade the packages, and maybe revert/pin the ones that don't have quick fixes. That's the difference. Once a given package is released/packaged for distribution, I want dependencies to be fixed so that every new installation works, and doesn't fail if some developer decided to break things that day.