3 ms·

Pip alone is not enough. Python makes backwards incompatible changes to the language all the time, even in dot releases. This means that you can't just install

Pip alone is not enough. Python makes backwards incompatible changes to the language all the time, even in dot releases. This means that you can't just install everything into some shared directories since some code would need version X of Python and some would need version Y. So you need virtualenv or something like that. But virtualenv isn't enough either because most actually useful Python code calls into C libraries (Pandas, Tensorflow, Matplotlib, etc.) And those need to be managed by something like RPM, deb, or even docker. This is leaving aside implementation issues. For example, many python dependency management tools are incredibly slow and there are some differences of opinion on how to express dependencies.

Epa095 4y ago
I don't understand your paragraph about C libraries. With wheels (which pip can install) there is no problem packing, and linking to, compiled C libraries.
- jujube3 4y ago
  It's often not feasible for legal or regulatory reasons to install bundled versions of C libraries. Like most enterprise software wants to use the system version of openssl because otherwise the vendor has to scramble to update for each openssl security issue. Even when it is possible to install bundled libraries, someone has to do the work to set up Python packages that bundle the native libraries. This work is done for some of the most important pacakges like TensorFlow, but probably not for some more esoteric library you want to use. I also suspect that people seriously using TensorFlow probably want careful control over which version they are using, rather than letting a random Python package manage that. Basically, interacting with native libraries is a hard problem to fully solve for any language. A lot of other languages like Java have struggled with this as well. But at least in Java you seldom use native dependencies. In Python you use them all the time (arguably, serving as glue code for crusty old FORTRAN and C libraries is where Python shines.) Python's stubborn determination not to standardize on anything for packaging (we have easy_install, conda, pip, pipenv, poetry, and who knows how many others) as well as its insistence on breaking its own compatibility certainly don't help either. And Python is more hostile than most languages to integrating with the system package manager (RPM, deb, etc.) because of how it sprays its files around the filesystem.

Epa095 4y ago

I don't understand your paragraph about C libraries. With wheels (which pip can install) there is no problem packing, and linking to, compiled C libraries.

jujube3 4y ago
It's often not feasible for legal or regulatory reasons to install bundled versions of C libraries. Like most enterprise software wants to use the system version of openssl because otherwise the vendor has to scramble to update for each openssl security issue. Even when it is possible to install bundled libraries, someone has to do the work to set up Python packages that bundle the native libraries. This work is done for some of the most important pacakges like TensorFlow, but probably not for some more esoteric library you want to use. I also suspect that people seriously using TensorFlow probably want careful control over which version they are using, rather than letting a random Python package manage that. Basically, interacting with native libraries is a hard problem to fully solve for any language. A lot of other languages like Java have struggled with this as well. But at least in Java you seldom use native dependencies. In Python you use them all the time (arguably, serving as glue code for crusty old FORTRAN and C libraries is where Python shines.) Python's stubborn determination not to standardize on anything for packaging (we have easy_install, conda, pip, pipenv, poetry, and who knows how many others) as well as its insistence on breaking its own compatibility certainly don't help either. And Python is more hostile than most languages to integrating with the system package manager (RPM, deb, etc.) because of how it sprays its files around the filesystem.