4 ms·

Asking as a complete newcomer to Python programming... Why doesn't somebody just fix pip? Or is pipenv just understood to be the fix?

jopython 4y ago
Second that. Pip has been the default for more than a decade. And yes it does have issues. Why not fix it or add options to pip instead of piling on more tools?
- bandrami 4y ago
  Because for whatever reason Python really attracts the Start Over And This Time We’ll Get It Right crowd. It’s why I don’t start new projects with it anymore, but make most of my living cleaning up the mess that incompatible changes make with existing software. So, in that sense, it’s great for keeping devops employed.
ehutch79 4y ago
There’s nothing really wrong with pip. It doesn’t have a lot of features, but its position isn’t one to have those features. It’s simple ina good way. I don’t need to futz with a bunch of stuff I don’t need to get it going
- rerdavies 4y ago
  Keras/tensorflow has some <= version specifications for modules it depends on. If there is a > version installed, pip install fails. Being able to install multiple versions of the same module seems like a fairly basic feature for a packaging manager. And it would seem to me that having a shim executable to examine a profile of some kind for an app at startup, and load/install the correct version of python doesn't seem impossible.
  - ehutch79 4y ago
    I find having multiple versions weird. That’s gotta be a niche need. If nothing else it seems like a good way to let packages be lax in keeping their dependencies up to date.
jujube3 4y ago
Pip alone is not enough. Python makes backwards incompatible changes to the language all the time, even in dot releases. This means that you can't just install everything into some shared directories since some code would need version X of Python and some would need version Y. So you need virtualenv or something like that. But virtualenv isn't enough either because most actually useful Python code calls into C libraries (Pandas, Tensorflow, Matplotlib, etc.) And those need to be managed by something like RPM, deb, or even docker. This is leaving aside implementation issues. For example, many python dependency management tools are incredibly slow and there are some differences of opinion on how to express dependencies.
- Epa095 4y ago
  I don't understand your paragraph about C libraries. With wheels (which pip can install) there is no problem packing, and linking to, compiled C libraries.
  - jujube3 4y ago
    It's often not feasible for legal or regulatory reasons to install bundled versions of C libraries. Like most enterprise software wants to use the system version of openssl because otherwise the vendor has to scramble to update for each openssl security issue. Even when it is possible to install bundled libraries, someone has to do the work to set up Python packages that bundle the native libraries. This work is done for some of the most important pacakges like TensorFlow, but probably not for some more esoteric library you want to use. I also suspect that people seriously using TensorFlow probably want careful control over which version they are using, rather than letting a random Python package manage that. Basically, interacting with native libraries is a hard problem to fully solve for any language. A lot of other languages like Java have struggled with this as well. But at least in Java you seldom use native dependencies. In Python you use them all the time (arguably, serving as glue code for crusty old FORTRAN and C libraries is where Python shines.) Python's stubborn determination not to standardize on anything for packaging (we have easy_install, conda, pip, pipenv, poetry, and who knows how many others) as well as its insistence on breaking its own compatibility certainly don't help either. And Python is more hostile than most languages to integrating with the system package manager (RPM, deb, etc.) because of how it sprays its files around the filesystem.

jopython 4y ago

Second that. Pip has been the default for more than a decade. And yes it does have issues. Why not fix it or add options to pip instead of piling on more tools?

bandrami 4y ago
Because for whatever reason Python really attracts the Start Over And This Time We’ll Get It Right crowd. It’s why I don’t start new projects with it anymore, but make most of my living cleaning up the mess that incompatible changes make with existing software. So, in that sense, it’s great for keeping devops employed.

ehutch79 4y ago

There’s nothing really wrong with pip. It doesn’t have a lot of features, but its position isn’t one to have those features. It’s simple ina good way. I don’t need to futz with a bunch of stuff I don’t need to get it going

rerdavies 4y ago
Keras/tensorflow has some <= version specifications for modules it depends on. If there is a > version installed, pip install fails. Being able to install multiple versions of the same module seems like a fairly basic feature for a packaging manager. And it would seem to me that having a shim executable to examine a profile of some kind for an app at startup, and load/install the correct version of python doesn't seem impossible.
- ehutch79 4y ago
  I find having multiple versions weird. That’s gotta be a niche need. If nothing else it seems like a good way to let packages be lax in keeping their dependencies up to date.

jujube3 4y ago

Pip alone is not enough. Python makes backwards incompatible changes to the language all the time, even in dot releases. This means that you can't just install everything into some shared directories since some code would need version X of Python and some would need version Y. So you need virtualenv or something like that. But virtualenv isn't enough either because most actually useful Python code calls into C libraries (Pandas, Tensorflow, Matplotlib, etc.) And those need to be managed by something like RPM, deb, or even docker. This is leaving aside implementation issues. For example, many python dependency management tools are incredibly slow and there are some differences of opinion on how to express dependencies.

Epa095 4y ago
I don't understand your paragraph about C libraries. With wheels (which pip can install) there is no problem packing, and linking to, compiled C libraries.
- jujube3 4y ago
  It's often not feasible for legal or regulatory reasons to install bundled versions of C libraries. Like most enterprise software wants to use the system version of openssl because otherwise the vendor has to scramble to update for each openssl security issue. Even when it is possible to install bundled libraries, someone has to do the work to set up Python packages that bundle the native libraries. This work is done for some of the most important pacakges like TensorFlow, but probably not for some more esoteric library you want to use. I also suspect that people seriously using TensorFlow probably want careful control over which version they are using, rather than letting a random Python package manage that. Basically, interacting with native libraries is a hard problem to fully solve for any language. A lot of other languages like Java have struggled with this as well. But at least in Java you seldom use native dependencies. In Python you use them all the time (arguably, serving as glue code for crusty old FORTRAN and C libraries is where Python shines.) Python's stubborn determination not to standardize on anything for packaging (we have easy_install, conda, pip, pipenv, poetry, and who knows how many others) as well as its insistence on breaking its own compatibility certainly don't help either. And Python is more hostile than most languages to integrating with the system package manager (RPM, deb, etc.) because of how it sprays its files around the filesystem.