6 ms·

Virtualenv's bin/activate is Doing It Wrong

107 points by kefeizhou 14y ago

timtadh 14y ago
I handle this issue (and a bunch of other environment variable issues) with a little utility I wrote. https://github.com/timtadh/swork https://github.com/timtadh/swork It basically figures out the pid of your shell and dumps the environment variables into a file in /tmp/swork. Then you make custom activate scripts for all of your projects. It has really helped me deal with a lot of projects. It also has convenience functions for quickly cd'ing to any project. When you want to get back to the original configuration it simply restores the original environment vars.
heykiddos 14y ago
This is also how all HPC computing environments (such as SLURM, SGE) do it. They simply spawn a subshell for you to work with and do whatever you want to run.
- michaelhoffman 14y ago
  As a long-time SGE user you are not describing something familiar to me. Can you describe what you mean more specifically?
tbatterii 14y ago
I haven't used activate in years, I just call the interpreter directly.
- thraxil 14y ago
  Yep. I do mostly Django development and my project template just automatically sets up a virtualenv in a 've' subdirectory and sets the shebang line in manage.py to "#!ve/bin/python". From there, I just run "./manage.py whatever" and I never even have to think about it.
  - untothebreach 14y ago
    that's a good way...since I already use Makefiles to do little tasks in my projects, I end up just setting `PYTHON=.virtualenv/bin/python` and in my targets, using `$(PYTHON)` instead of `python`. Ditto for `pip`, `nosetests`, etc.
regularfry 14y ago
Ah, I like this. I'm going to cavort off with these ideas and put them to good use in my rubygems environment switcher gemenv.
- regularfry 14y ago
  ...aaaaand done. At least, the subshell launcher part. https://github.com/regularfry/gemenv https://github.com/regularfry/gemenv, if anyone 'round these parts is interested.
deleted 14y ago
[deleted]
cabalamat 14y ago
The approach i use is simply: $ cat run #!/bin/sh . venv/bin/activate python main.py Is this wrong? If so, what would work better?
- ibejoeb 14y ago
  Not wrong, but if you're just running a program under a virtualenv (as opposed to doing shell work), just: /path/to/env/bin/python main.py will suffice. No need to create a separate shell script, start a new shell, and activate. Great for cronjobs.
diminoten 14y ago
None of these problems are the kinds of problems I've ever run into. They're entirely a series of what if and edge case scenarios in which virtualenv is not the answer, which is fine. I don't need a tool that can do things I'm never going to do.
snprbob86 14y ago
This is also the same technique I like to use for providing configuration to applications. Basically, write YOUR_APP_ENV=prod or whatever into /etc/environment and then add appenv to PATH which tests against the machine's environment, sets all the appropriate environment variables accordingly and ends with an `exec`. Super nice way to provide shell-friendly language-agnostic configuration. I used to use `exec env $@` at the end of the script, so that I could `appenv > prod` for diffing, but I actually prefer this articles suggestion: `exec "${@:-$SHELL}"` since you can still `appenv env > prod`, but `appenv` provides an interactive shell by default.
bpatrianakos 14y ago
Virtualenv bin/activate isn't doing anything "wrong". It's just not doing it how the author believes is best. Can the whole "You're doing X wrong" meme die already please? I like the idea though! From what I see so far the author makes a good argument for using a subshell rather than setting and resetting environment variables. At the same time though, I haven't enough experience with his suggestion yet to see the downsides... and there most definitely will be. Why? Because that's programming, man. There's no panacea, no perfect way to do things, no true "best" way to do anything. So is this the "right" way? Probably not. Better is probably more like it and using that word instead makes you seem, you know, like not cocky asshole (not that I think that about the author at all, just generally speaking).
- jrockway 14y ago
  Can the whole "You're doing X wrong" meme die already please? You clicked through to an article you wouldn't otherwise read. The meme persists because it works, and complaining about it just makes it more effective. (I didn't click through because I don't really care if X is doing Y wrong.)
  - bpatrianakos 14y ago
    It does work and it makes me hate it even more! Because by the time something really is wrong I'll be immune to the meme and I'll never know what's wrong.
- kenko 14y ago
  "Virtualenv bin/activate isn't doing anything "wrong". It's just not doing it how the author believes is best." So is it possible to do something wrong, or is it only possible to do something that another person (or you) believes is not best? "So is this the "right" way? Probably not. Better is probably more like it ..." So this way is better, but the other way wasn't wrong? How could this way be better?
amckinlay 14y ago
...and Python 3.3 just included virtualenv as a standard library...
lifeisstillgood 14y ago
Venv is one of the pieces of sanity in the python packaging deployment world and any improvements welcome - this goes on my List if things to think hard about
Comkid 14y ago
Reading the Google Groups responses, he's officially created a http://xkcd.com/1172/ http://xkcd.com/1172/ situation heh
- avdd_ 14y ago
  I assume you mean this thread https://groups.google.com/d/topic/python-virtualenv/XzI8GStvKDw/discussion https://groups.google.com/d/topic/python-virtualenv/XzI8GStv...
kennu 14y ago
But what will be the added CO2 footprint of launching so many unnecessary shells! Also, I don't like the prospect of having to hit Ctrl-D twice to logout.
- simonh 14y ago
  > But what will be the added CO2 footprint of launching so many unnecessary shells! The shells are actually fairly easy to recycle responsibly, it's fork()s I'm worried about.
- __alexs 14y ago
  Then type 'exec inve' instead of just 'inve.'
  - kennu 14y ago
    Perhaps that makes sense. Actually I usually type 'workon myproject' (using virtualenvwrapper, because it provides a nice standard location and activation command for virtualenvs), so maybe this could all be transparent.
the_mitsuhiko 14y ago
The fact that it does not use a subshell makes it useful. You can trivially use it in bash scripts and cronjobs.
- ajross 14y ago
  Because "eval `myscript`" is somehow non-trivial? No, this is just bad. The whole paradigm of having build setup depend on local variables in an interactive shell (which is pervasive, virtualenv is hardly the only bad actor here) is just broken inherently. It's lazy programming. It creates all sorts of failure modes -- builds can magically stop working when people update their .bashrc files (e.g. set PATH explicitly, but OOPS the build system expects to source a script and then spawn you an interactive shell), etc... Implicit configuration is bad, OK? Just don't do it.
- IgorPartola 14y ago
  Why can you not do?: #!/usr/bin/virtual-bash my-env echo hello Edit: in fact something like /usr/bin/virt-python my-env is much nicer than /var/lib/my-app/env/bin/python. Basically like virtualenvwrapper but for Python scripts that are not run interactively.
  - the_mitsuhiko 14y ago
    I regularly write scripts that involve more than one virtualenv. At the moment this is trivial by just activating and deactivating envs from one script. Starting subshells is complicated because communication into the shell barely exists. Worse than that is that the shebang only supports two arguments and is heavily length limited.
    - IgorPartola 14y ago
      Unless you are rapidly switching envs, you could launch the parts of your scripts that operate on different envs as subscripts. Then you can pipe data in and out. Can you provide an example where this does not work? Point taken on the shebang limitations.

107 points by kefeizhou 14y ago

I handle this issue (and a bunch of other environment variable issues) with a little utility I wrote. https://github.com/timtadh/swork https://github.com/timtadh/swork It basically figures out the pid of your shell and dumps the environment variables into a file in /tmp/swork. Then you make custom activate scripts for all of your projects. It has really helped me deal with a lot of projects. It also has convenience functions for quickly cd'ing to any project. When you want to get back to the original configuration it simply restores the original environment vars.

heykiddos 14y ago

This is also how all HPC computing environments (such as SLURM, SGE) do it. They simply spawn a subshell for you to work with and do whatever you want to run.

michaelhoffman 14y ago
As a long-time SGE user you are not describing something familiar to me. Can you describe what you mean more specifically?

tbatterii 14y ago

I haven't used activate in years, I just call the interpreter directly.

thraxil 14y ago
Yep. I do mostly Django development and my project template just automatically sets up a virtualenv in a 've' subdirectory and sets the shebang line in manage.py to "#!ve/bin/python". From there, I just run "./manage.py whatever" and I never even have to think about it.
- untothebreach 14y ago
  that's a good way...since I already use Makefiles to do little tasks in my projects, I end up just setting `PYTHON=.virtualenv/bin/python` and in my targets, using `$(PYTHON)` instead of `python`. Ditto for `pip`, `nosetests`, etc.

regularfry 14y ago

Ah, I like this. I'm going to cavort off with these ideas and put them to good use in my rubygems environment switcher gemenv.

regularfry 14y ago
...aaaaand done. At least, the subshell launcher part. https://github.com/regularfry/gemenv https://github.com/regularfry/gemenv, if anyone 'round these parts is interested.

deleted 14y ago

[deleted]

cabalamat 14y ago

The approach i use is simply: $ cat run #!/bin/sh . venv/bin/activate python main.py Is this wrong? If so, what would work better?

ibejoeb 14y ago
Not wrong, but if you're just running a program under a virtualenv (as opposed to doing shell work), just: /path/to/env/bin/python main.py will suffice. No need to create a separate shell script, start a new shell, and activate. Great for cronjobs.

diminoten 14y ago

None of these problems are the kinds of problems I've ever run into. They're entirely a series of what if and edge case scenarios in which virtualenv is not the answer, which is fine. I don't need a tool that can do things I'm never going to do.

snprbob86 14y ago

This is also the same technique I like to use for providing configuration to applications. Basically, write YOUR_APP_ENV=prod or whatever into /etc/environment and then add appenv to PATH which tests against the machine's environment, sets all the appropriate environment variables accordingly and ends with an `exec`. Super nice way to provide shell-friendly language-agnostic configuration. I used to use `exec env $@` at the end of the script, so that I could `appenv > prod` for diffing, but I actually prefer this articles suggestion: `exec "${@:-$SHELL}"` since you can still `appenv env > prod`, but `appenv` provides an interactive shell by default.

bpatrianakos 14y ago

Virtualenv bin/activate isn't doing anything "wrong". It's just not doing it how the author believes is best. Can the whole "You're doing X wrong" meme die already please? I like the idea though! From what I see so far the author makes a good argument for using a subshell rather than setting and resetting environment variables. At the same time though, I haven't enough experience with his suggestion yet to see the downsides... and there most definitely will be. Why? Because that's programming, man. There's no panacea, no perfect way to do things, no true "best" way to do anything. So is this the "right" way? Probably not. Better is probably more like it and using that word instead makes you seem, you know, like not cocky asshole (not that I think that about the author at all, just generally speaking).

jrockway 14y ago
Can the whole "You're doing X wrong" meme die already please? You clicked through to an article you wouldn't otherwise read. The meme persists because it works, and complaining about it just makes it more effective. (I didn't click through because I don't really care if X is doing Y wrong.)
- bpatrianakos 14y ago
  It does work and it makes me hate it even more! Because by the time something really is wrong I'll be immune to the meme and I'll never know what's wrong.
kenko 14y ago
"Virtualenv bin/activate isn't doing anything "wrong". It's just not doing it how the author believes is best." So is it possible to do something wrong, or is it only possible to do something that another person (or you) believes is not best? "So is this the "right" way? Probably not. Better is probably more like it ..." So this way is better, but the other way wasn't wrong? How could this way be better?

amckinlay 14y ago

...and Python 3.3 just included virtualenv as a standard library...

lifeisstillgood 14y ago

Venv is one of the pieces of sanity in the python packaging deployment world and any improvements welcome - this goes on my List if things to think hard about

Comkid 14y ago

Reading the Google Groups responses, he's officially created a http://xkcd.com/1172/ http://xkcd.com/1172/ situation heh

avdd_ 14y ago
I assume you mean this thread https://groups.google.com/d/topic/python-virtualenv/XzI8GStvKDw/discussion https://groups.google.com/d/topic/python-virtualenv/XzI8GStv...

kennu 14y ago

But what will be the added CO2 footprint of launching so many unnecessary shells! Also, I don't like the prospect of having to hit Ctrl-D twice to logout.

simonh 14y ago
> But what will be the added CO2 footprint of launching so many unnecessary shells! The shells are actually fairly easy to recycle responsibly, it's fork()s I'm worried about.
__alexs 14y ago
Then type 'exec inve' instead of just 'inve.'
- kennu 14y ago
  Perhaps that makes sense. Actually I usually type 'workon myproject' (using virtualenvwrapper, because it provides a nice standard location and activation command for virtualenvs), so maybe this could all be transparent.

the_mitsuhiko 14y ago

The fact that it does not use a subshell makes it useful. You can trivially use it in bash scripts and cronjobs.

ajross 14y ago
Because "eval `myscript`" is somehow non-trivial? No, this is just bad. The whole paradigm of having build setup depend on local variables in an interactive shell (which is pervasive, virtualenv is hardly the only bad actor here) is just broken inherently. It's lazy programming. It creates all sorts of failure modes -- builds can magically stop working when people update their .bashrc files (e.g. set PATH explicitly, but OOPS the build system expects to source a script and then spawn you an interactive shell), etc... Implicit configuration is bad, OK? Just don't do it.
IgorPartola 14y ago
Why can you not do?: #!/usr/bin/virtual-bash my-env echo hello Edit: in fact something like /usr/bin/virt-python my-env is much nicer than /var/lib/my-app/env/bin/python. Basically like virtualenvwrapper but for Python scripts that are not run interactively.
- the_mitsuhiko 14y ago
  I regularly write scripts that involve more than one virtualenv. At the moment this is trivial by just activating and deactivating envs from one script. Starting subshells is complicated because communication into the shell barely exists. Worse than that is that the shebang only supports two arguments and is heavily length limited.
  - IgorPartola 14y ago
    Unless you are rapidly switching envs, you could launch the parts of your scripts that operate on different envs as subscripts. Then you can pipe data in and out. Can you provide an example where this does not work? Point taken on the shebang limitations.