4 ms·

The next Debian/Ubuntu releases will no longer allow `pip install` outside of a venv: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-e

by lazka 4y ago

The next Debian/Ubuntu releases will no longer allow `pip install` outside of a venv: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/69 https://discuss.python.org/t/pep-668-marking-python-base-env... You can still force it via `pip install --break-system-packages ...` if needed.

meitham 4y ago
Hopefully that’s not going to be the case inside a container!
- forgotpwd16 4y ago
  Why? Can overwrite it and even if couldn't making a new venv is just a `python -m venv venv` away.
  - qbasic_forever 4y ago
    Venv in a container is unnecessary ritual. It's a container, it has its own entire root filesystem...
    - pxc 4y ago
      You could have a container whose entry point is a shell script that calls multiple Python programs that need different environments, or a multiprocess container that runs multiple Python programs, although I guess you could still address either by breaking down your containers differently.
wheelerof4te 4y ago
The way it's meant to be. On Linux, you either use the system packages via "apt install", or you use venvs. EDIT: For context, I've meant "managed" distros like Debian and Ubuntu.
- qbasic_forever 4y ago
  Nowhere in the official Python documentation (where 99% of new python users are going to go) does it warn or even talk about Linux and Debian specific issues like only using apt packaged versions of dependencies. It wasn't even until recent years that pip gave a hint or warning something might break in those setups. The situation with Python on Debian has been pretty bad IMHO with a cloistered group of people saying the status quo is just fine because it works for them exclusively.
  - wheelerof4te 4y ago
    "The situation with Python on Debian has been pretty bad IMHO" The situation with anything has been pretty bad in Debian lately. I'm all for the minimalistic approach in regards to Python. It's Ok to provide only the packages needed by applications and the core system. For everything else, there's pip. EDIT: I've meant to say, there's pip inside a venv.
    - qbasic_forever 4y ago
      I agree, the problem is if you just 'pip install foo' on Debian, like 99% of python docs and readmes say, it instantly fails. Worse yet if you think you should just elevate to root and force pip install to work you can potentially break the Python install. It's a nasty footgun.
unxdfa 4y ago
This makes me so happy. Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Different pipelines would be running in different requirements files. I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) and reprovisioned the whole build cluster resulting in pipelines breaking consistently where they should have been breaking: trying to do stupid stuff. Personally I only ever use the system python packages on Linux if I can get away with it. Saves a whole world of problems.
- KronisLV 4y ago
  > Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Not everyone might like containers, but using them for CI seems like a good way to avoid situations like this, at least when viable (e.g. web development). You get to choose what container images you need for your build, do whatever is necessary inside of them and they're essentially thrown away after once you're done with what you need to do, cache aside. They also don't have any significant impact or dependencies on the system configuration either, as long as you have some sort of a supported container runtime.
  - kkielhofner 4y ago
    Second containers (however controversial they may otherwise be). The official python Alpine container is < 20mb and the slim Debian variants are < 45mb. For many of my Python projects I end up needing to install various system dependency libs like CUDA, libav, libsndfile, etc. I tend to be a "fan" of containers generally and containers seem like the best way to handle situations like mine.
- hnfong 4y ago
  > I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) If you allowed sudo in your jenkins jobs you're morally barred from blaming python for screwing up the system.
ziml77 4y ago
Mixing pip with another package manager has always seemed weird to me. You're just asking for things to conflict and break. I noticed with Homebrew that there was no way to untangle packages installed through pip and ones installed through Homebrew. After dealing with that mess once, I now make sure to use pip install --user. It can still cause things to break, but if that does happen it's at least easy to nuke the packages installed to my home directory.
deleted 4y ago
[deleted]
TheRealPomax 4y ago
Good. Now we just need to get pip itself updated so it refuses to run outside of a venv, and refuses to run unless invoked with "python -m pip" and we'll finally have something at least half decent. And don't even get me started about how much better npm is at publishing packages, versus pip's refusal to add the same user friendliness.
Groxx 4y ago
This is fantastic to hear. Hopefully this will be the beginning of a wave of other OSes doing the same. For anyone on other systems who wants this kind of protection right now, pip has had this available for a few years at least: pip config set global.require-virtualenv True I absolutely recommend doing it. Immediately.
ianwootten 4y ago
PEP 704 is a recent proposal to require a virtual environment by default for any package installer - https://peps.python.org/pep-0704/ https://peps.python.org/pep-0704/. Again, you can opt-out if you want.

by lazka 4y ago

meitham 4y ago

Hopefully that’s not going to be the case inside a container!

forgotpwd16 4y ago
Why? Can overwrite it and even if couldn't making a new venv is just a `python -m venv venv` away.
- qbasic_forever 4y ago
  Venv in a container is unnecessary ritual. It's a container, it has its own entire root filesystem...
  - pxc 4y ago
    You could have a container whose entry point is a shell script that calls multiple Python programs that need different environments, or a multiprocess container that runs multiple Python programs, although I guess you could still address either by breaking down your containers differently.

wheelerof4te 4y ago

The way it's meant to be. On Linux, you either use the system packages via "apt install", or you use venvs. EDIT: For context, I've meant "managed" distros like Debian and Ubuntu.

qbasic_forever 4y ago
Nowhere in the official Python documentation (where 99% of new python users are going to go) does it warn or even talk about Linux and Debian specific issues like only using apt packaged versions of dependencies. It wasn't even until recent years that pip gave a hint or warning something might break in those setups. The situation with Python on Debian has been pretty bad IMHO with a cloistered group of people saying the status quo is just fine because it works for them exclusively.
- wheelerof4te 4y ago
  "The situation with Python on Debian has been pretty bad IMHO" The situation with anything has been pretty bad in Debian lately. I'm all for the minimalistic approach in regards to Python. It's Ok to provide only the packages needed by applications and the core system. For everything else, there's pip. EDIT: I've meant to say, there's pip inside a venv.
  - qbasic_forever 4y ago
    I agree, the problem is if you just 'pip install foo' on Debian, like 99% of python docs and readmes say, it instantly fails. Worse yet if you think you should just elevate to root and force pip install to work you can potentially break the Python install. It's a nasty footgun.

unxdfa 4y ago

This makes me so happy. Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Different pipelines would be running in different requirements files. I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) and reprovisioned the whole build cluster resulting in pipelines breaking consistently where they should have been breaking: trying to do stupid stuff. Personally I only ever use the system python packages on Linux if I can get away with it. Saves a whole world of problems.

KronisLV 4y ago
> Back when we had Jenkins slaves, one of our devops guys set a pipeline up that pip installed different versions over the top of system packages causing weird intermittent failures everywhere. Not everyone might like containers, but using them for CI seems like a good way to avoid situations like this, at least when viable (e.g. web development). You get to choose what container images you need for your build, do whatever is necessary inside of them and they're essentially thrown away after once you're done with what you need to do, cache aside. They also don't have any significant impact or dependencies on the system configuration either, as long as you have some sort of a supported container runtime.
- kkielhofner 4y ago
  Second containers (however controversial they may otherwise be). The official python Alpine container is < 20mb and the slim Debian variants are < 45mb. For many of my Python projects I end up needing to install various system dependency libs like CUDA, libav, libsndfile, etc. I tend to be a "fan" of containers generally and containers seem like the best way to handle situations like mine.
hnfong 4y ago
> I revoked sudo privs immediately for Jenkins (I didn't add them in the first place) If you allowed sudo in your jenkins jobs you're morally barred from blaming python for screwing up the system.

ziml77 4y ago

Mixing pip with another package manager has always seemed weird to me. You're just asking for things to conflict and break. I noticed with Homebrew that there was no way to untangle packages installed through pip and ones installed through Homebrew. After dealing with that mess once, I now make sure to use pip install --user. It can still cause things to break, but if that does happen it's at least easy to nuke the packages installed to my home directory.

deleted 4y ago

[deleted]

TheRealPomax 4y ago

Good. Now we just need to get pip itself updated so it refuses to run outside of a venv, and refuses to run unless invoked with "python -m pip" and we'll finally have something at least half decent. And don't even get me started about how much better npm is at publishing packages, versus pip's refusal to add the same user friendliness.

Groxx 4y ago

This is fantastic to hear. Hopefully this will be the beginning of a wave of other OSes doing the same. For anyone on other systems who wants this kind of protection right now, pip has had this available for a few years at least: pip config set global.require-virtualenv True I absolutely recommend doing it. Immediately.

ianwootten 4y ago

PEP 704 is a recent proposal to require a virtual environment by default for any package installer - https://peps.python.org/pep-0704/ https://peps.python.org/pep-0704/. Again, you can opt-out if you want.