7 ms·

PEP 704 – Require virtual environments by default for package installers

60 points by BiteCode_dev 4y ago

BiteCode_dev 4y ago
One of the biggest problem in python packaging is... not packaging. Bootstrapping a workable dev env is hard: you can fail at installing python, setting it up, creating a venv, activating it... This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv. First, it can break things. But also, it will probably not work, and if it does, it will bite them later on. So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user. Ideally, we should have something like cargo or npm, which solve the entire problem altogether, but believe it or not, it's not easy to do for a project as old and successful as python. This is a healthy compromise while something better is decided.
- adql 4y ago
  > This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv. And even more "just someone that wants to install python app" > So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user. I'd like if the default was just "create venv for user with default name and put it there". So the above case of "user that just wants to install app" just works, and the crap pip spews is nicely contained.
  - BiteCode_dev 4y ago
    We cannot do that yet because not all systems come with venv install, thanks to linux.
Apreche 4y ago
What would be even better would be to completely remove the requirement to activate or deactivate the virtual environments. If someone runs python it should look for a .venv directory in the current working directory. If it finds one, it should use the python interpreter located in that virtual environment instead of the system one. Obviously a flag can be used to disable this behavior and run the system interpreter no matter what the current directory is. There’s an argument to be made that it should search upwards in the directory tree as well. In case the current directory is /home/user/myproject/otherdir/deeperdir/ and the .venv is in /home/user/myproject/.venv/ then the venv should still be used. This not only saves a lot of trouble for users in the terminal, but also for various shell scripts, systemd service definitions, etc. Anyone who wants to run a Python program correctly just has to make sure they cd to the correct place first. Then they can just run python without having to know anything at all about virtualenvs.
- whalesalad 4y ago
  This defeats one of the coolest parts of virtualenvs which is that they can be placed anywhere. It’s just a directory.
  - tedivm 4y ago
    This pep already partially does that. > The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named .venv, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
  - meandmycode 4y ago
    Why do you say this is one of the coolest part?
    - whalesalad 4y ago
      Because the best part about unix/linux is that everything is a file (or more broadly, everything are files and folders) and this is congruent with that. The more I program, the more I appreciate: - 1. tools that are extremely small and functional and can be composed together (re: unix pipes, cat foo | grep bar) - 2. things that are just files and folders on disk, so that they can get introspected, moved to different disks, symlinked, put on FUSE filesystems, etc. The common interface of file or folder is very flexible.
PaulHoule 4y ago
It’s about time. Other than “pip’s solving strategy is unsound” there are a handful of problems that make Python environments unreliable and this is one of the worst of them. If eggs go away the “pip is unsound” problem will be easier to solve.
TekMol 4y ago
I run Python applications in containers. No need for a virtual environment. Don't you need a container anyhow? Who want's any Python program and its 1000 dependencies to have access to the guest OS?
- substation13 4y ago
  Containers can be broken out of. They are not a security tool. However, from a configuration perspective I totally agree. Python ecosystem is broken beyond repair. Just chuck it all in Docker and get back to programming!
  - TekMol 4y ago
    Containers can be broken out of. Every time I hear that, I challenge the person to prove it. So far, nobody did.
    - substation13 4y ago
      Did you ask a skilled pen-tester?
jjice 4y ago
I'm sure there's a reason that I'm just not aware of, but why is the PEP for __pypackages__ [1] in draft and not the obvious solution? It just seems that having those dependencies in the directory would resolve a good few issues without the need for the virtual environment. The big downside would be versioning of the Python interpreter, but at that point, we could still keep virtual environments for that case, right? I assume I'm missing something big and someone who is more in-sync with the Python ecosystem can correct me. [1] https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/
- qbasic_forever 4y ago
  Too many cooks in the kitchen. There's a ton of competing ideas to 'fix' packaging in python, but no will or desire from the core to step up and own/shepherd a solution. There's a recent podcast with most of the folks important to the ecosystem and they pretty quickly get to the realization that yes it sucks for users but it's not their domain and no one really owns or can fix python packaging: https://talkpython.fm/episodes/show/406/reimagining-pythons-packaging-workflows https://talkpython.fm/episodes/show/406/reimagining-pythons-...
zitterbewegung 4y ago
I would really like this for the single reason that while there are many virtual environment or similar systems I see this as still a win for the beginner but, still having the flexibility of using other systems. Honestly, having multiple virtual environment like systems is very confusing. I generally use condo and the venv module but I couldn't understand how to get poetry to work.
- suslik 4y ago
  > I couldn't understand how to get poetry to work. I recommend looking into it again. Conda is great for managing software environments, but when developing pure python packages, poetry is brilliant.
noirscape 4y ago
Can't say I'm a fan of this one. I get the idea, but I don't like the execution of this. It's a very myopic view of python package management; python packages are also often just used to distribute complete applications (ie. youtube-dl), not just libraries. This is one of the reasons why pip nowadays warns you if you do a sudo installation to site-packages and otherwise defaults to user; binaries need to be installed to the right place and all this will likely end up doing is further mangling the amount of PATH fiddling necessary to get those packages working. I also don't like that we're getting yet another expected reserve name for virtual environments. Originally it was virtualenv, then it became venv and now it could end up being .venv. Python's package management could use quite a bit of work, especially with versioned dependencies co-existing, and venvs are a useful solution to that but this isn't the answer.
tyingq 4y ago
I feel like this proposal underestimates the size of the audience that doesn't want, need, or use python's various virtual environment options.
- Flimm 4y ago
  How would they be affected by this, either positively or negatively? The people who don't use virtual environments tend not to use pip either.
Euphorbium 4y ago
This is great, and I would make sure to move to 3.13+ because of this.
jononomo 4y ago
Why is Python the only language for which this is such a headache?
twen_ty 4y ago
Now give me relocatable virtual environments and I'm game
tupolef 4y ago
I always hated updates between python3 versions, with or without venvs, but I still installed many packages with pip from time to time because it was a reflex. But Debian is currently pushing for this new requirement and it made me discover PIPX, which is a PIP including venvs and python version upgrades without any hassle. Now I use APT or PIPX all the time.

60 points by BiteCode_dev 4y ago

One of the biggest problem in python packaging is... not packaging. Bootstrapping a workable dev env is hard: you can fail at installing python, setting it up, creating a venv, activating it... This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv. First, it can break things. But also, it will probably not work, and if it does, it will bite them later on. So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user. Ideally, we should have something like cargo or npm, which solve the entire problem altogether, but believe it or not, it's not easy to do for a project as old and successful as python. This is a healthy compromise while something better is decided.

adql 4y ago
> This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv. And even more "just someone that wants to install python app" > So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user. I'd like if the default was just "create venv for user with default name and put it there". So the above case of "user that just wants to install app" just works, and the crap pip spews is nicely contained.
- BiteCode_dev 4y ago
  We cannot do that yet because not all systems come with venv install, thanks to linux.

Apreche 4y ago

What would be even better would be to completely remove the requirement to activate or deactivate the virtual environments. If someone runs python it should look for a .venv directory in the current working directory. If it finds one, it should use the python interpreter located in that virtual environment instead of the system one. Obviously a flag can be used to disable this behavior and run the system interpreter no matter what the current directory is. There’s an argument to be made that it should search upwards in the directory tree as well. In case the current directory is /home/user/myproject/otherdir/deeperdir/ and the .venv is in /home/user/myproject/.venv/ then the venv should still be used. This not only saves a lot of trouble for users in the terminal, but also for various shell scripts, systemd service definitions, etc. Anyone who wants to run a Python program correctly just has to make sure they cd to the correct place first. Then they can just run python without having to know anything at all about virtualenvs.

whalesalad 4y ago
This defeats one of the coolest parts of virtualenvs which is that they can be placed anywhere. It’s just a directory.
- tedivm 4y ago
  This pep already partially does that. > The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named .venv, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
- meandmycode 4y ago
  Why do you say this is one of the coolest part?
  - whalesalad 4y ago
    Because the best part about unix/linux is that everything is a file (or more broadly, everything are files and folders) and this is congruent with that. The more I program, the more I appreciate: - 1. tools that are extremely small and functional and can be composed together (re: unix pipes, cat foo | grep bar) - 2. things that are just files and folders on disk, so that they can get introspected, moved to different disks, symlinked, put on FUSE filesystems, etc. The common interface of file or folder is very flexible.

PaulHoule 4y ago

It’s about time. Other than “pip’s solving strategy is unsound” there are a handful of problems that make Python environments unreliable and this is one of the worst of them. If eggs go away the “pip is unsound” problem will be easier to solve.

TekMol 4y ago

I run Python applications in containers. No need for a virtual environment. Don't you need a container anyhow? Who want's any Python program and its 1000 dependencies to have access to the guest OS?

substation13 4y ago
Containers can be broken out of. They are not a security tool. However, from a configuration perspective I totally agree. Python ecosystem is broken beyond repair. Just chuck it all in Docker and get back to programming!
- TekMol 4y ago
  Containers can be broken out of. Every time I hear that, I challenge the person to prove it. So far, nobody did.
  - substation13 4y ago
    Did you ask a skilled pen-tester?

jjice 4y ago

I'm sure there's a reason that I'm just not aware of, but why is the PEP for __pypackages__ [1] in draft and not the obvious solution? It just seems that having those dependencies in the directory would resolve a good few issues without the need for the virtual environment. The big downside would be versioning of the Python interpreter, but at that point, we could still keep virtual environments for that case, right? I assume I'm missing something big and someone who is more in-sync with the Python ecosystem can correct me. [1] https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/

qbasic_forever 4y ago
Too many cooks in the kitchen. There's a ton of competing ideas to 'fix' packaging in python, but no will or desire from the core to step up and own/shepherd a solution. There's a recent podcast with most of the folks important to the ecosystem and they pretty quickly get to the realization that yes it sucks for users but it's not their domain and no one really owns or can fix python packaging: https://talkpython.fm/episodes/show/406/reimagining-pythons-packaging-workflows https://talkpython.fm/episodes/show/406/reimagining-pythons-...

zitterbewegung 4y ago

I would really like this for the single reason that while there are many virtual environment or similar systems I see this as still a win for the beginner but, still having the flexibility of using other systems. Honestly, having multiple virtual environment like systems is very confusing. I generally use condo and the venv module but I couldn't understand how to get poetry to work.

suslik 4y ago
> I couldn't understand how to get poetry to work. I recommend looking into it again. Conda is great for managing software environments, but when developing pure python packages, poetry is brilliant.

noirscape 4y ago

Can't say I'm a fan of this one. I get the idea, but I don't like the execution of this. It's a very myopic view of python package management; python packages are also often just used to distribute complete applications (ie. youtube-dl), not just libraries. This is one of the reasons why pip nowadays warns you if you do a sudo installation to site-packages and otherwise defaults to user; binaries need to be installed to the right place and all this will likely end up doing is further mangling the amount of PATH fiddling necessary to get those packages working. I also don't like that we're getting yet another expected reserve name for virtual environments. Originally it was virtualenv, then it became venv and now it could end up being .venv. Python's package management could use quite a bit of work, especially with versioned dependencies co-existing, and venvs are a useful solution to that but this isn't the answer.

tyingq 4y ago

I feel like this proposal underestimates the size of the audience that doesn't want, need, or use python's various virtual environment options.

Flimm 4y ago
How would they be affected by this, either positively or negatively? The people who don't use virtual environments tend not to use pip either.

Euphorbium 4y ago

This is great, and I would make sure to move to 3.13+ because of this.

jononomo 4y ago

Why is Python the only language for which this is such a headache?

twen_ty 4y ago

Now give me relocatable virtual environments and I'm game

tupolef 4y ago

I always hated updates between python3 versions, with or without venvs, but I still installed many packages with pip from time to time because it was a reflex. But Debian is currently pushing for this new requirement and it made me discover PIPX, which is a PIP including venvs and python version upgrades without any hassle. Now I use APT or PIPX all the time.