3 ms·

You have to get these things right early-on, and get everyone on the same page. Once things start to get fragmented, once you have historical decisions that pre

You have to get these things right early-on, and get everyone on the same page. Once things start to get fragmented, once you have historical decisions that prevent you from improving certain aspects, etc, there's only so much you can do to after the fact to paper over the gaps. Python's packaging story is rotten in its bones. I think at this point it's nearly impossible to fix (though many continue to try). The way I see it, a solution would require: - A cohesive, self-contained, easy to use, all-inclusive toolset, with a completely standardized manifest format for packages. And, for all that is holy, doesn't mess with the global environment so that we have to lie to it with virtual environments. - That toolset would have to get overwhelming adoption by the community, to where it retroactively becomes The Python Package Manager (outside of legacy codebases, which would continue to languish). This would probably require endorsement by the Python developers, or for the tool to be so unassailably better that it's preferred without exception, or possibly both. Otherwise we'll continue to have: https://xkcd.com/927/ https://xkcd.com/927/ I want to emphasize that on the second point 50% adoption is not enough. 70% adoption is not enough. I'm talking 90%+ adoption for libraries and new projects; everything outside of it needs to be a rounding error, or we're still in fragmentation hell. And then even in the best case- there would be a long tail for years to come of packages that haven't been ported to the new tool/manifest format, where you have to duck out to one of the other package managers to pull them in.

majou 4y ago
https://xkcd.com/1987/ https://xkcd.com/1987/
hardwaregeek 4y ago
Yeah at this point it appears to be a cultural issue. There’s been plenty of opportunity to anoint a proper package manager that doesn’t require virtual environments or install things globally by default, that uses lock files and resolves dependencies without too much fuss. If I’m not mistaken poetry and pipenv offer this. But there just isn’t the buy in. People in the Python world, including many in this thread, just seem to refuse this path and insist, proudly, that their workflow is just fine. It’s a kind of psychological curiosity that people would resist proper package management so fervently.
- PaulHoule 4y ago
  The average data scientist thinks it is OK to have a manual process to make the monthly report so long as it takes less than 1 month and so does their manager. And so does the VC that funds it. I mean, you can’t write a command line program which is reliable and straightforward to install in blub, why would anybody need it?
melony 4y ago
There's also the fact that Python is old. Python is used as Linux's unofficial system scripting language the same way Macs use Ruby. I think we should mandate a SYSTEM_PYTHON variable to avoid polluting the PYTHONPATH further with brittle global installs that can't be easily upgraded or replaced.
- rcarmo 4y ago
  Erm. Brew uses Ruby, and there are historical reasons about that, but Apple never had any favorites in that race.
BerislavLopac 4y ago
> - A cohesive, self-contained, easy to use, all-inclusive toolset, The only complex thing about Python packaging is that is separates it tools into "frontend" and "backend" build tools. On the frontend, pip is essentially the cohesive, self-contained, easy to use, all-inclusive solution, as long as your backends support the standards [0][1]. > with a completely standardized manifest format for packages That's wheel [2]. Everything else is deprecated now, except for sdist (which doesn't really matter as it's limited to Python-only packages, and transparently supported by all tools). > And, for all that is holy, doesn't mess with the global environment so that we have to lie to it with virtual environments. This would be nice, but it is completely out of hands of the language developers. Different Linux (and other OSs, like OSX) distributions have made different decisions about the python version(s) they bundle by default, and in many cases the bundled versions are critical for some OS-level tools to function properly. Pretty much all interpreted languages either a) have their versions of virtual environments, or b) rely on a single system-wide interpreter. That being said, there is no need to use virtualenvs if a) you know that you need only one version of Python, and b) you have full control over your system. The best example is service deployments, where you can install a version of your choosing (or start with an appropriate base image), and install everything else on top of it. > That toolset would have to get overwhelming adoption by the community, to where it retroactively becomes The Python Package Manager (outside of legacy codebases, which would continue to languish). This would probably require endorsement by the Python developers Why exactly? There are two ways one can work with packages: 1. The majority of users want to be able to install packages. This is already the case. Pip is overwhelmingly adopted by everyone, with two notable exceptions: * Conda, which is mainly used in some niche sub-communities, and which is fully compatible with pip anyway * Build tools being used for installing packages, either during development or for deployment in very controlled environments But both of these exceptions are very specific and narrow in scope. > Otherwise we'll continue to have: https://xkcd.com/927/ https://xkcd.com/927/ We already don't have that situation. I agree that we did, but the standards are now very clear, and the tools are slowly moving in the right direction. [0] https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [1] https://peps.python.org/pep-0660/ https://peps.python.org/pep-0660/ [2] https://realpython.com/python-wheels/ https://realpython.com/python-wheels/

majou 4y ago

https://xkcd.com/1987/ https://xkcd.com/1987/

hardwaregeek 4y ago

Yeah at this point it appears to be a cultural issue. There’s been plenty of opportunity to anoint a proper package manager that doesn’t require virtual environments or install things globally by default, that uses lock files and resolves dependencies without too much fuss. If I’m not mistaken poetry and pipenv offer this. But there just isn’t the buy in. People in the Python world, including many in this thread, just seem to refuse this path and insist, proudly, that their workflow is just fine. It’s a kind of psychological curiosity that people would resist proper package management so fervently.

PaulHoule 4y ago
The average data scientist thinks it is OK to have a manual process to make the monthly report so long as it takes less than 1 month and so does their manager. And so does the VC that funds it. I mean, you can’t write a command line program which is reliable and straightforward to install in blub, why would anybody need it?

melony 4y ago

There's also the fact that Python is old. Python is used as Linux's unofficial system scripting language the same way Macs use Ruby. I think we should mandate a SYSTEM_PYTHON variable to avoid polluting the PYTHONPATH further with brittle global installs that can't be easily upgraded or replaced.

rcarmo 4y ago
Erm. Brew uses Ruby, and there are historical reasons about that, but Apple never had any favorites in that race.

BerislavLopac 4y ago

> - A cohesive, self-contained, easy to use, all-inclusive toolset, The only complex thing about Python packaging is that is separates it tools into "frontend" and "backend" build tools. On the frontend, pip is essentially the cohesive, self-contained, easy to use, all-inclusive solution, as long as your backends support the standards [0][1]. > with a completely standardized manifest format for packages That's wheel [2]. Everything else is deprecated now, except for sdist (which doesn't really matter as it's limited to Python-only packages, and transparently supported by all tools). > And, for all that is holy, doesn't mess with the global environment so that we have to lie to it with virtual environments. This would be nice, but it is completely out of hands of the language developers. Different Linux (and other OSs, like OSX) distributions have made different decisions about the python version(s) they bundle by default, and in many cases the bundled versions are critical for some OS-level tools to function properly. Pretty much all interpreted languages either a) have their versions of virtual environments, or b) rely on a single system-wide interpreter. That being said, there is no need to use virtualenvs if a) you know that you need only one version of Python, and b) you have full control over your system. The best example is service deployments, where you can install a version of your choosing (or start with an appropriate base image), and install everything else on top of it. > That toolset would have to get overwhelming adoption by the community, to where it retroactively becomes The Python Package Manager (outside of legacy codebases, which would continue to languish). This would probably require endorsement by the Python developers Why exactly? There are two ways one can work with packages: 1. The majority of users want to be able to install packages. This is already the case. Pip is overwhelmingly adopted by everyone, with two notable exceptions: * Conda, which is mainly used in some niche sub-communities, and which is fully compatible with pip anyway * Build tools being used for installing packages, either during development or for deployment in very controlled environments But both of these exceptions are very specific and narrow in scope. > Otherwise we'll continue to have: https://xkcd.com/927/ https://xkcd.com/927/ We already don't have that situation. I agree that we did, but the standards are now very clear, and the tools are slowly moving in the right direction. [0] https://peps.python.org/pep-0518/ https://peps.python.org/pep-0518/ [1] https://peps.python.org/pep-0660/ https://peps.python.org/pep-0660/ [2] https://realpython.com/python-wheels/ https://realpython.com/python-wheels/