6 ms·

What would be even better would be to completely remove the requirement to activate or deactivate the virtual environments. If someone runs python it should lo

What would be even better would be to completely remove the requirement to activate or deactivate the virtual environments. If someone runs python it should look for a .venv directory in the current working directory. If it finds one, it should use the python interpreter located in that virtual environment instead of the system one. Obviously a flag can be used to disable this behavior and run the system interpreter no matter what the current directory is. There’s an argument to be made that it should search upwards in the directory tree as well. In case the current directory is /home/user/myproject/otherdir/deeperdir/ and the .venv is in /home/user/myproject/.venv/ then the venv should still be used. This not only saves a lot of trouble for users in the terminal, but also for various shell scripts, systemd service definitions, etc. Anyone who wants to run a Python program correctly just has to make sure they cd to the correct place first. Then they can just run python without having to know anything at all about virtualenvs.

whalesalad 4y ago
This defeats one of the coolest parts of virtualenvs which is that they can be placed anywhere. It’s just a directory.
- tedivm 4y ago
  This pep already partially does that. > The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named .venv, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
- meandmycode 4y ago
  Why do you say this is one of the coolest part?
  - whalesalad 4y ago
    Because the best part about unix/linux is that everything is a file (or more broadly, everything are files and folders) and this is congruent with that. The more I program, the more I appreciate: - 1. tools that are extremely small and functional and can be composed together (re: unix pipes, cat foo | grep bar) - 2. things that are just files and folders on disk, so that they can get introspected, moved to different disks, symlinked, put on FUSE filesystems, etc. The common interface of file or folder is very flexible.
- twic 4y ago
  Nothing about that proposal changes how venvs work. It just changes what the global python command does. Anything you do explicitly with a venv today, you could still do under that proposal.
- DangitBobby 4y ago
  I'd call this a minor convenience at best. Symlinks would still make it possible, of course.
  - whalesalad 4y ago
    I know it's not the end of the world, but I prefer to keep my dependencies outside of the project root in deployed scenarios. Ex, /srv/venv and /srv/app
PaulHoule 4y ago
I like activate setting the path. “python3”, “pip3” is the best idea anyone ever got since Solomon suggested cutting a baby in half. I mean, soon it becomes “python3.5” (don’t really try that one!), “python3.6”, “python3.7”, “python3.8”, and maybe “python3.9-pypi” and who knows what else. If the noobs you are pandering are too lazy to create a venv, they will never be able to change your script that says pip3.12.8a-lyrical-magical install xyzyzy To match pip3.14q or whatever version they are running. With the venv you are much closer to a write-once, run-anywhere situation for writing Python backed and supporting scripts.
qbasic_forever 4y ago
That's more or less what PEP 582 plans to do, but it's been stalled and mired in discussions for years. The PDM tool went ahead and implemented it though if you want to use it: https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm
milliams 4y ago
There is not an actual need to activate for most use cases. I almost always just do python3 -m venv .venv .venv/bin/pip install whatever .venv/bin/python foo.py
- nerdponx 4y ago
  This is what I do in 99% of my projects. Or I use a tool like Hatch that creates envs for me, and I can use `hatch run ...` instead of manually invoking the path. People like to complain about Python, but `./venv/bin/foo` is basically identical to `./node_modules/.bin/foo` and I don't see why the former deserves so much hate.
  - gymbeaux 4y ago
    I think it's because you "have to" run a command to activate the virtual environment AKA use files that are already in the directory. With NodeJS (or any other platform/language), it's implied that the files in your current directory are the files you want to use for building/execution
- twic 4y ago
  Explicit is better than implicit, after all.
- tomjakubowski 4y ago
  Just to give an example where activate is needed, if you have the tool programs `a` and `b` installed in the venv, and `a` invokes `b` (invokes as opposed to importing python modules), then: .venv/bin/a wouldn't be able to invoke b without activating the venv or otherwise putting .venv/bin on the PATH.
drcongo 4y ago
I have my Fish shell do this automatically when I cd into a directory with a .venv
- WirelessGigabit 4y ago
  Which plugin do you use?
  - drcongo 4y ago
    I don't use a plugin, just a custom fish function that responds to directory changes. Mine is quite simple but also very specific to our systems, however this one looks very solid: https://gist.github.com/tommyip/cf9099fa6053e30247e5d0318de2fb9e https://gist.github.com/tommyip/cf9099fa6053e30247e5d0318de2...
mattgreenrocks 4y ago
Written more Python than I'd like, and I still hate + avoid venvs. They're purely incidental complexity: it should be able to set up a project-local venv by themselves without me having to micromanage anything. I don't care about activating/deactivating venvs. It should just work when I navigate to the project folder. It should let me tweak where packages are stored for the 1% of the time that I care.
- radicalbyte 4y ago
  It always seems so backwards to me when I use Python... and kinda crazy to see in a "beginner" language.
  - throwaway894345 4y ago
    I've always felt that it's weird that Python is considered a "beginner" language. Yeah, it uses "and" instead of "&&", but it took me a loooonnng time to figure out how to manage dependencies when I was starting out (and then figuring out how to deploy to targets, how to do reproducible builds, etc). Compared with Rust or Go which just do the right thing by default (personally I've always felt that the tooling is the way more important problem than language concepts which often feel like bike shedding to me).
    - radicalbyte 4y ago
      I also despair that it's used as a beginner language. There are so many better options. I even include Java in that list. So why is it used? As it has a fantastic eco-system for applied statistics / machine learning / various math things. If you're teaching an introduction to programming then it's a very effective language. That and once people progress it's decent - much prefer the code written in Python by competent developers to that written in Java or even Go. It also sets people up to use more robust languages such as C# or (for those with a Math background) Haskell.
- bitcurious 4y ago
  >It should just work when I navigate to the project folder. You still need to do the setup when you create the project, but once you do you can use direnv[0] to activate/deactivate the venv[1]. I target GCP often, so I have direnv switch over to the correct project [3] also. [0] https://direnv.net/ https://direnv.net/ [1] https://stackabuse.com/managing-python-environments-with-direnv-and-pyenv/ https://stackabuse.com/managing-python-environments-with-dir... [2] https://stackoverflow.com/questions/56781493/gcloud-is-it-possible-to-change-the-default-project-based-on-the-current-folder https://stackoverflow.com/questions/56781493/gcloud-is-it-po...
ptx 4y ago
Loading code from the current working directory is not a good idea. It would make it unsafe to run a Python program from any directory that's not completely trusted, e.g. if we imagine "ls" having this feature: [root@localhost]# cd /home/someguy [root@localhost]# ls Oops, now someguy can execute arbitrary code as root. Windows used to do this with DLL files and has since had to add various modes where it's either disabled or (in the default "safe DLL search mode") deprioritized in the search order: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic...
- qbasic_forever 4y ago
  That's not the proposal, the idea with PEP 582 is python's executable will check a local directory for a venv and activate it if necessary, but only when you actually run python (not automatically in your shell).
  - ptx 4y ago
    I was responding to the parent comment which suggested that Python "should look for a .venv directory in the current working directory". Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected.
    - coldtea 4y ago
      >Doing this "only when you actually run python" doesn't help Node.js does it this way, and it works just fine. Not sure what you think the problem is. >Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected Why would the current working directory matter as to that? First, if somebody has write access to your local directory you have worse issues. They could for example overwrite the py/pyc dependencies or replace the python inside the virtual env, and their code will be executed if you run any of the items in the bin or even import those python deps. Whether Python "looks for a .venv directory in the current working directory" or not doesn't mattter. It will just mean that you need to activate the .venv first to get those in your PATH. But that's why you had the .venv there in the first place, to activate and use it. Whether you activate it manually or not doesn't matter...
gymbeaux 4y ago
Correct. This PEP makes sense if venvs are easier to work with
noirscape 4y ago
This would be a mess with python updates fwiw. venvs in my experience break when the system version of python is updated. By solving it this way, you open a whole other can of issues.
twic 4y ago
At some point i did this with gradle (i'm a Java guy): https://hg.sr.ht/~twic/devtools/browse/bin/gradle?rev=tip https://hg.sr.ht/~twic/devtools/browse/bin/gradle?rev=tip If i'm in a directory with a gradle wrapper, i essentially never want to run any globally installed gradle, i want to run the one in the wrapper. It's more of a mixed bag with python. If i do: cd ~/myproject python main.py Then yes, i want to use the venv in ~/myproject. If i do: cd ~/myproject salt '*' status.all_status I do not want to run salt using the venv in ~/myproject! On the other hand, if i do: python ~/myproject/main.py I think i'd like to use the venv in ~/myproject. So maybe resolve the file that will get run, then look for a venv in the vicinity of that? What happens when you do: cd ~/myproject python -m json.tool data.json ? Another option would be to introduce a new set of commands - vpython runs python using a nearby venv, vpip runs pip using it, etc. Or call the ppython, ppip, etc - less mnemonic, but faster to type.
andybak 4y ago
Virtualenvwrapper used to do this. It worked pretty flawlessly if I recall. Not doing much Python now so I don't know what the current situation is.
avnigo 4y ago
But you can have multiple virtual environments in the same directory, which one would it activate? You could also be using an environment that's in a different directory, or have one already loaded. I like the concept potentially meaning less friction, but the implementation may make things more confusing for beginners when you're not explicitly activating an environment in your shell. I think maybe a flag to enable the behavior at runtime would be better than it'd be opt out.
- jdxcode 4y ago
  .venv, obviously. Just like node and node_modules. No need to make this configurable beyond VIRTUAL_ENV
- coldtea 4y ago
  >But you can have multiple virtual environments in the same directory, which one would it activate? So don't have many in the same directory. Since the name shall be fixed, you wouldn't be able to anyway (e.g. .venv, similar to .node_modules). Simple as that.
deleted 4y ago
[deleted]
Maphistow 4y ago
Different tool but: https://asdf-vm.com/ https://asdf-vm.com/ Works like this
BiteCode_dev 4y ago
This is in discussion, but is not straighforward. We need a story for system python which does not want this behavior, a security story so that an attacker cannot just pop a dir and make you execute it, a story for entry points, a story for parent env handling and so on. pypackages is a proposal for this, but since it's harder to do, and this one easier, one step after the other...

whalesalad 4y ago

This defeats one of the coolest parts of virtualenvs which is that they can be placed anywhere. It’s just a directory.

tedivm 4y ago
This pep already partially does that. > The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named .venv, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
meandmycode 4y ago
Why do you say this is one of the coolest part?
- whalesalad 4y ago
  Because the best part about unix/linux is that everything is a file (or more broadly, everything are files and folders) and this is congruent with that. The more I program, the more I appreciate: - 1. tools that are extremely small and functional and can be composed together (re: unix pipes, cat foo | grep bar) - 2. things that are just files and folders on disk, so that they can get introspected, moved to different disks, symlinked, put on FUSE filesystems, etc. The common interface of file or folder is very flexible.
twic 4y ago
Nothing about that proposal changes how venvs work. It just changes what the global python command does. Anything you do explicitly with a venv today, you could still do under that proposal.
DangitBobby 4y ago
I'd call this a minor convenience at best. Symlinks would still make it possible, of course.
- whalesalad 4y ago
  I know it's not the end of the world, but I prefer to keep my dependencies outside of the project root in deployed scenarios. Ex, /srv/venv and /srv/app

PaulHoule 4y ago

I like activate setting the path. “python3”, “pip3” is the best idea anyone ever got since Solomon suggested cutting a baby in half. I mean, soon it becomes “python3.5” (don’t really try that one!), “python3.6”, “python3.7”, “python3.8”, and maybe “python3.9-pypi” and who knows what else. If the noobs you are pandering are too lazy to create a venv, they will never be able to change your script that says pip3.12.8a-lyrical-magical install xyzyzy To match pip3.14q or whatever version they are running. With the venv you are much closer to a write-once, run-anywhere situation for writing Python backed and supporting scripts.

qbasic_forever 4y ago

That's more or less what PEP 582 plans to do, but it's been stalled and mired in discussions for years. The PDM tool went ahead and implemented it though if you want to use it: https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm

milliams 4y ago

There is not an actual need to activate for most use cases. I almost always just do python3 -m venv .venv .venv/bin/pip install whatever .venv/bin/python foo.py

nerdponx 4y ago
This is what I do in 99% of my projects. Or I use a tool like Hatch that creates envs for me, and I can use `hatch run ...` instead of manually invoking the path. People like to complain about Python, but `./venv/bin/foo` is basically identical to `./node_modules/.bin/foo` and I don't see why the former deserves so much hate.
- gymbeaux 4y ago
  I think it's because you "have to" run a command to activate the virtual environment AKA use files that are already in the directory. With NodeJS (or any other platform/language), it's implied that the files in your current directory are the files you want to use for building/execution
twic 4y ago
Explicit is better than implicit, after all.
tomjakubowski 4y ago
Just to give an example where activate is needed, if you have the tool programs `a` and `b` installed in the venv, and `a` invokes `b` (invokes as opposed to importing python modules), then: .venv/bin/a wouldn't be able to invoke b without activating the venv or otherwise putting .venv/bin on the PATH.

drcongo 4y ago

I have my Fish shell do this automatically when I cd into a directory with a .venv

WirelessGigabit 4y ago
Which plugin do you use?
- drcongo 4y ago
  I don't use a plugin, just a custom fish function that responds to directory changes. Mine is quite simple but also very specific to our systems, however this one looks very solid: https://gist.github.com/tommyip/cf9099fa6053e30247e5d0318de2fb9e https://gist.github.com/tommyip/cf9099fa6053e30247e5d0318de2...

mattgreenrocks 4y ago

Written more Python than I'd like, and I still hate + avoid venvs. They're purely incidental complexity: it should be able to set up a project-local venv by themselves without me having to micromanage anything. I don't care about activating/deactivating venvs. It should just work when I navigate to the project folder. It should let me tweak where packages are stored for the 1% of the time that I care.

radicalbyte 4y ago
It always seems so backwards to me when I use Python... and kinda crazy to see in a "beginner" language.
- throwaway894345 4y ago
  I've always felt that it's weird that Python is considered a "beginner" language. Yeah, it uses "and" instead of "&&", but it took me a loooonnng time to figure out how to manage dependencies when I was starting out (and then figuring out how to deploy to targets, how to do reproducible builds, etc). Compared with Rust or Go which just do the right thing by default (personally I've always felt that the tooling is the way more important problem than language concepts which often feel like bike shedding to me).
  - radicalbyte 4y ago
    I also despair that it's used as a beginner language. There are so many better options. I even include Java in that list. So why is it used? As it has a fantastic eco-system for applied statistics / machine learning / various math things. If you're teaching an introduction to programming then it's a very effective language. That and once people progress it's decent - much prefer the code written in Python by competent developers to that written in Java or even Go. It also sets people up to use more robust languages such as C# or (for those with a Math background) Haskell.
bitcurious 4y ago
>It should just work when I navigate to the project folder. You still need to do the setup when you create the project, but once you do you can use direnv[0] to activate/deactivate the venv[1]. I target GCP often, so I have direnv switch over to the correct project [3] also. [0] https://direnv.net/ https://direnv.net/ [1] https://stackabuse.com/managing-python-environments-with-direnv-and-pyenv/ https://stackabuse.com/managing-python-environments-with-dir... [2] https://stackoverflow.com/questions/56781493/gcloud-is-it-possible-to-change-the-default-project-based-on-the-current-folder https://stackoverflow.com/questions/56781493/gcloud-is-it-po...

ptx 4y ago

Loading code from the current working directory is not a good idea. It would make it unsafe to run a Python program from any directory that's not completely trusted, e.g. if we imagine "ls" having this feature: [root@localhost]# cd /home/someguy [root@localhost]# ls Oops, now someguy can execute arbitrary code as root. Windows used to do this with DLL files and has since had to add various modes where it's either disabled or (in the default "safe DLL search mode") deprioritized in the search order: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic...

qbasic_forever 4y ago
That's not the proposal, the idea with PEP 582 is python's executable will check a local directory for a venv and activate it if necessary, but only when you actually run python (not automatically in your shell).
- ptx 4y ago
  I was responding to the parent comment which suggested that Python "should look for a .venv directory in the current working directory". Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected.
  - coldtea 4y ago
    >Doing this "only when you actually run python" doesn't help Node.js does it this way, and it works just fine. Not sure what you think the problem is. >Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected Why would the current working directory matter as to that? First, if somebody has write access to your local directory you have worse issues. They could for example overwrite the py/pyc dependencies or replace the python inside the virtual env, and their code will be executed if you run any of the items in the bin or even import those python deps. Whether Python "looks for a .venv directory in the current working directory" or not doesn't mattter. It will just mean that you need to activate the .venv first to get those in your PATH. But that's why you had the .venv there in the first place, to activate and use it. Whether you activate it manually or not doesn't matter...

gymbeaux 4y ago

Correct. This PEP makes sense if venvs are easier to work with

noirscape 4y ago

This would be a mess with python updates fwiw. venvs in my experience break when the system version of python is updated. By solving it this way, you open a whole other can of issues.

twic 4y ago

At some point i did this with gradle (i'm a Java guy): https://hg.sr.ht/~twic/devtools/browse/bin/gradle?rev=tip https://hg.sr.ht/~twic/devtools/browse/bin/gradle?rev=tip If i'm in a directory with a gradle wrapper, i essentially never want to run any globally installed gradle, i want to run the one in the wrapper. It's more of a mixed bag with python. If i do: cd ~/myproject python main.py Then yes, i want to use the venv in ~/myproject. If i do: cd ~/myproject salt '*' status.all_status I do not want to run salt using the venv in ~/myproject! On the other hand, if i do: python ~/myproject/main.py I think i'd like to use the venv in ~/myproject. So maybe resolve the file that will get run, then look for a venv in the vicinity of that? What happens when you do: cd ~/myproject python -m json.tool data.json ? Another option would be to introduce a new set of commands - vpython runs python using a nearby venv, vpip runs pip using it, etc. Or call the ppython, ppip, etc - less mnemonic, but faster to type.

andybak 4y ago

Virtualenvwrapper used to do this. It worked pretty flawlessly if I recall. Not doing much Python now so I don't know what the current situation is.

avnigo 4y ago

But you can have multiple virtual environments in the same directory, which one would it activate? You could also be using an environment that's in a different directory, or have one already loaded. I like the concept potentially meaning less friction, but the implementation may make things more confusing for beginners when you're not explicitly activating an environment in your shell. I think maybe a flag to enable the behavior at runtime would be better than it'd be opt out.

jdxcode 4y ago
.venv, obviously. Just like node and node_modules. No need to make this configurable beyond VIRTUAL_ENV
coldtea 4y ago
>But you can have multiple virtual environments in the same directory, which one would it activate? So don't have many in the same directory. Since the name shall be fixed, you wouldn't be able to anyway (e.g. .venv, similar to .node_modules). Simple as that.

deleted 4y ago

[deleted]

Maphistow 4y ago

Different tool but: https://asdf-vm.com/ https://asdf-vm.com/ Works like this

BiteCode_dev 4y ago

This is in discussion, but is not straighforward. We need a story for system python which does not want this behavior, a security story so that an attacker cannot just pop a dir and make you execute it, a story for entry points, a story for parent env handling and so on. pypackages is a proposal for this, but since it's harder to do, and this one easier, one step after the other...