4 ms·

I have yet to come across a scenario where installing a python library as global ended up being an issue in the python ecosystem. You either need it, or you don

by zo1 10y ago

I have yet to come across a scenario where installing a python library as global ended up being an issue in the python ecosystem. You either need it, or you don't. Short of some contrived scenario where a piece of code is not able to utilize a certain package's latest code and you have to juggle two different versions of that package. If I had something like that, I'd fix the code, not juggle old package versions or install all dependencies in my project's sub-folder. > "For example, why does Python packaging need such a complicated bootstrapping process? Install ensure_pip, then the real pip, and on top of that virtualenv." It's not that complicated: sudo apt-get install python-pip python-dev build-essential sudo pip install --upgrade pip Then you can install any python package to your heart's content with: pip install PACKAGE You don't need virtualenv, nor do you need ensurepip. >"In the Python/PIP world, everything is central and even today you still have to accept extra hassle to get a sane, local setup." I think you may be coming at it from a javascript perspective where you expect it to work according to the way you're used to it working with Node/NPM. I have absolutely no hassle installing anything in the python/pip ecosystem. Except maybe MySQL and OpenCV, I'll grant you that. But those are separate discussions as they require external dependencies outside of python's packaging.

Eridrus 10y ago
I've run into several issues with global installs because the Ubuntu maintainers have decided to make changes to libraries I want to use.
- gcr 10y ago
  Another example happens when compiling scientific libraries that require C extensions. I've seen disagreements about which versions of libraries to link against. For example, I've run into issues when installing scikit-learn via pip because it linked against the Numpy version in /usr/lib rather than the correct Numpy in my virtualenv.
- rlpb 10y ago
  Ubuntu maintainer here. Can you be more specific, please? I'd like to understand if there was a general reason or if we're doing something that we should not be doing.
  - Eridrus 10y ago
    Appologies for my crappy memory, but the only case where I can remember the package was when I ran into issues with the PIL library being OS-managed and having basic functionality not working, and the answer I found that worked was installing it in a virtualenv. I also went through some chat logs of me complaining and I saw I had similar issues with urllib3. The main issue is that they're modified at all, so they don't match up with the docs or what people say online, so until you realise that you're not running the normal code and need to run inside a virtualenv it's very confusing.
    - rlpb 10y ago
      The only changes made to PIL in Ubuntu in any current release (Precise, Trusty, Xenial) are security fixes: https://launchpad.net/ubuntu/+source/pillow https://launchpad.net/ubuntu/+source/pillow. > The main issue is that they're modified at all Not modifying packages at all would also be a major issue. We'd end up shipping dysfunctional packages, or packages with major bugs, or not fixing bugs during the lifetime of a release. Distributions do a ton of integration work that I think goes unnoticed by users. We routinely find issues and send them upstream - every development cycle. Users don't notice because they'll also find the bugfixes in upstream releases because we sent them there. But upstream releases don't all happen in lock-step, so we end up having to carry some patches to make everything work with everything else. In the general case, we can't win. We'll always upset someone, which is why I asked for specifics. In general, we will: (0) Avoid changing anything in a stable release, so users don't have things changed on them - except for (1). (1) Fix bugs (including security bugs) as they are reported. Because of (0), this generally means that we cherry-pick fixes, which does cause some divergence from upstream. But if we don't do this, then other users complain either about the bugs or, if we pull everything into a package in an existing stable release, about things being changed under them. (2) During development of a release we fix for integration issues, since every package in the distribution is expected to work with every other package - and (though with some exceptions) we generally only ship one version of each thing in a release. This means that we sometimes need to patch things (we prefer cherry-picks from upstream, but sometimes we don't get a response from upstream in time for our release schedule) in order to make everything work. But in the specific case of PIL, we appear to not have had to do any of these things, except for some security fixes. For urllib3, it looks like we had to make some modifications to fix security issues with regards to TLS and certificate verification in Trusty, and one bug in Xenial related to broken IPv6 square bracket handling, and that's it. I don't really see how we can do anything different here. Should we have ignored the security issues and shipped vulnerable packages? Or should we have upset many more users of the stable release by bumping behaviour under the feet of existing deployments?
kgrin 10y ago
Here's a common scenario: you maintain more than one project, and you can't just simultaneously upgrade e.g. Django for all of them (or literally any library where an upgrade may be non-trivial). Yes, eventually you'll (hopefully) upgrade all your projects to the newest and shiniest versions of all your dependencies, but if you need to maintain some semblance of stability, that's not always immediately possible/practical.
- m_mueller 10y ago
  I'd rather use docker over venv in that case. My host python is for horsing around, when something is to be deployed I document and isolate the dependencies in a dockerfile. venv would only solve half that problem and only for python, not including outside delendencies.
  - jjnoakes 10y ago
    Docker doesn't work if you don't have root.
    - mikewhy 10y ago
      You need to have access to the docker socket, if you're using sockets to connect. That's not necessarily only root.
      - jjnoakes 10y ago
        https://news.ycombinator.com/item?id=12291092 https://news.ycombinator.com/item?id=12291092
    - m_mueller 10y ago
      True. I also work in a HPC context where that's a problem. That's actually a place where I had to hack together something with venv in order to use an output analysis with a python that's compiled differently from the main Fortran executable. But that's quite an edge case.
    - 10y ago
zepolen 10y ago
Not everyone is running debian flavored os, in fact not everyone is running linux.
- RubyPinch 10y ago
  for windows, in powershell (Invoke-WebRequest -UseBasicParsing "https://bootstrap.pypa.io/get-pip.py").Content | python - but generally in that case, one doesn't need to install ensure_pip, and can use that. And one doesn't need to install pip at all, since it is an optional come-with but in most cases, the `curl url | python -` or equivalent solves the problem
vog 10y ago
> I think you may be coming at it from a javascript perspective where you expect it to work according to the way you're used to it working with Node/NPM Please avoid making such assumptions about other people. It's just a little step from there to ad hominems. To make my point more clear: I love Python and 'm used to Python/PIP, but when I had to work with NodeJS and saw NPM, I immediately thought: These people got it right. Not the language, mind you, but the package management!

by zo1 10y ago

Eridrus 10y ago

I've run into several issues with global installs because the Ubuntu maintainers have decided to make changes to libraries I want to use.

gcr 10y ago
Another example happens when compiling scientific libraries that require C extensions. I've seen disagreements about which versions of libraries to link against. For example, I've run into issues when installing scikit-learn via pip because it linked against the Numpy version in /usr/lib rather than the correct Numpy in my virtualenv.
rlpb 10y ago
Ubuntu maintainer here. Can you be more specific, please? I'd like to understand if there was a general reason or if we're doing something that we should not be doing.
- Eridrus 10y ago
  Appologies for my crappy memory, but the only case where I can remember the package was when I ran into issues with the PIL library being OS-managed and having basic functionality not working, and the answer I found that worked was installing it in a virtualenv. I also went through some chat logs of me complaining and I saw I had similar issues with urllib3. The main issue is that they're modified at all, so they don't match up with the docs or what people say online, so until you realise that you're not running the normal code and need to run inside a virtualenv it's very confusing.
  - rlpb 10y ago
    The only changes made to PIL in Ubuntu in any current release (Precise, Trusty, Xenial) are security fixes: https://launchpad.net/ubuntu/+source/pillow https://launchpad.net/ubuntu/+source/pillow. > The main issue is that they're modified at all Not modifying packages at all would also be a major issue. We'd end up shipping dysfunctional packages, or packages with major bugs, or not fixing bugs during the lifetime of a release. Distributions do a ton of integration work that I think goes unnoticed by users. We routinely find issues and send them upstream - every development cycle. Users don't notice because they'll also find the bugfixes in upstream releases because we sent them there. But upstream releases don't all happen in lock-step, so we end up having to carry some patches to make everything work with everything else. In the general case, we can't win. We'll always upset someone, which is why I asked for specifics. In general, we will: (0) Avoid changing anything in a stable release, so users don't have things changed on them - except for (1). (1) Fix bugs (including security bugs) as they are reported. Because of (0), this generally means that we cherry-pick fixes, which does cause some divergence from upstream. But if we don't do this, then other users complain either about the bugs or, if we pull everything into a package in an existing stable release, about things being changed under them. (2) During development of a release we fix for integration issues, since every package in the distribution is expected to work with every other package - and (though with some exceptions) we generally only ship one version of each thing in a release. This means that we sometimes need to patch things (we prefer cherry-picks from upstream, but sometimes we don't get a response from upstream in time for our release schedule) in order to make everything work. But in the specific case of PIL, we appear to not have had to do any of these things, except for some security fixes. For urllib3, it looks like we had to make some modifications to fix security issues with regards to TLS and certificate verification in Trusty, and one bug in Xenial related to broken IPv6 square bracket handling, and that's it. I don't really see how we can do anything different here. Should we have ignored the security issues and shipped vulnerable packages? Or should we have upset many more users of the stable release by bumping behaviour under the feet of existing deployments?

kgrin 10y ago

Here's a common scenario: you maintain more than one project, and you can't just simultaneously upgrade e.g. Django for all of them (or literally any library where an upgrade may be non-trivial). Yes, eventually you'll (hopefully) upgrade all your projects to the newest and shiniest versions of all your dependencies, but if you need to maintain some semblance of stability, that's not always immediately possible/practical.

m_mueller 10y ago
I'd rather use docker over venv in that case. My host python is for horsing around, when something is to be deployed I document and isolate the dependencies in a dockerfile. venv would only solve half that problem and only for python, not including outside delendencies.
- jjnoakes 10y ago
  Docker doesn't work if you don't have root.
  - mikewhy 10y ago
    You need to have access to the docker socket, if you're using sockets to connect. That's not necessarily only root.
    - jjnoakes 10y ago
      https://news.ycombinator.com/item?id=12291092 https://news.ycombinator.com/item?id=12291092
  - m_mueller 10y ago
    True. I also work in a HPC context where that's a problem. That's actually a place where I had to hack together something with venv in order to use an output analysis with a python that's compiled differently from the main Fortran executable. But that's quite an edge case.
  - 10y ago

zepolen 10y ago

Not everyone is running debian flavored os, in fact not everyone is running linux.

RubyPinch 10y ago
for windows, in powershell (Invoke-WebRequest -UseBasicParsing "https://bootstrap.pypa.io/get-pip.py").Content | python - but generally in that case, one doesn't need to install ensure_pip, and can use that. And one doesn't need to install pip at all, since it is an optional come-with but in most cases, the `curl url | python -` or equivalent solves the problem

vog 10y ago

> I think you may be coming at it from a javascript perspective where you expect it to work according to the way you're used to it working with Node/NPM Please avoid making such assumptions about other people. It's just a little step from there to ad hominems. To make my point more clear: I love Python and 'm used to Python/PIP, but when I had to work with NodeJS and saw NPM, I immediately thought: These people got it right. Not the language, mind you, but the package management!