9 ms·

A non-magical introduction to pip and virtualenv for Python beginners

241 points by j4mie 13y ago

rvasco 13y ago
Flawless explanation. As a somewhat beginner to python i must give my thanks !
netcraft 13y ago
a good introduction - I would like to hear more about deployment with virtualenv though - is it expected that you just document any packages with requirements.txt and then you would create the virtualenv in the deployment target and set everything up again? Or can you "package" a virtualenv for deployment?
- HeyImAlex 13y ago
  Just generate your requirements file from your env locally (pip freeze > requirements.txt), deploy all of your files (env folder excluded) to your sever however you want and then run 'env/bin/pip install -r requirements.txt' on your server.
  - turtle4 13y ago
    If I use pip freeze > requirements.txt, it lists the packages I have installed using pip, but it doesn't list anything for the python version itself. How do I make sure the right python version gets captured if I don't check in the /env/ folder?
- deleted 13y ago
  [deleted]
- fernly 13y ago
  Deployment/distribution can be handled by bundling the app. For Python 2.x, there's PyInstaller[0], Py2App[1], Py2Exe[2] all of which do much the same thing: make a single binary out of a python app and all its dependencies including the interpreter. Then you distribute that and don't worry about what the user has or hasn't got. [0]http://www.pyinstaller.org/ http://www.pyinstaller.org/ [1]https://pypi.python.org/pypi/py2app/ https://pypi.python.org/pypi/py2app/ [2]https://pypi.python.org/pypi/py2exe/0.6.9 https://pypi.python.org/pypi/py2exe/0.6.9
tocomment 13y ago
He mentions not checking the env directory into git. Why not? In general what are the best practices for using virtualenv with version control?
- afandian 13y ago
  What if you want to perform a security update for one of the libraries on the server? Also, IIRC the environment will contain a symlimk to the python executable and companion files. That symlink will change depending on the environment.
- kingrolo 13y ago
  Keep your requirements file checked in, but not the virtualenv. The built env should be seen as disposable, and is both location (ie, path on disk) and platform (for libs which are not pure python) specific . I keep my virtualenvs in ~/.virtualenvs/, away from the project.
- cinquemb 13y ago
  You could have issues with other people not running the same version of python, also you might have different site packages if you are working on an experimental branch that will be pushed or merged later.
- almost 13y ago
  Virtual envs don't relocate well, they tend to be very specific to machine and even install location. Plus you can recreate them from your requirements.txt file so there's no need. Just add the env directory to your .gitignore
- arnarbi 13y ago
  I find it best to keep virtual envs completely away from the project (I use http://virtualenvwrapper.readthedocs.org/en/latest/ http://virtualenvwrapper.readthedocs.org/en/latest/ which puts them by default in ~/.virtualenvs). A virtualenv is completely machine-specific. If your project is a package itself (i.e. it has a setup.py file), then use that file to specify dependencies. On a new machine I check out a copy, create a virtual env and activate it. Then in the local copy I run "pip install -e .". This installs all the requirements from setup.py in the virtualenv, and links the local copy of my project to it as well. Now your package is available in the virtual env, but fully editable. If your python project is not a package, you can install its dependencies in a virtual env with pip. Then run "pip freeze" to generate a list of all installed packages. Save that to a text file in your repository, e.g. ``requirements.txt``. On a different machine, or a fresh venv, you can then do "pip install -r requirements.txt" to set everything up in one go.
anonozdotcom 13y ago
My mentor did not even give such a good introduction. Kudos to the author, now I have a bright AHHA lightbulb in my head! :D
- deleted 13y ago
  [deleted]
- suyash 13y ago
  That is why they say to take multiple advice from multiple sources.
almost 13y ago
If you're in Brighton in the UK and you like this then maybe you'll be interested in the one day workshop run by Jamie (author of the blog post) and myself. This post was actually based on some of the material written by Jamie for the course. Next one is happening next Thursday and there are still a couple of tickets: http://dabapps.com/services/training/python-for-programmers http://dabapps.com/services/training/python-for-programmers
wmt 13y ago
"pip is vastly superior toeasy_install for lots of reasons, and so should generally be used instead." Unless you are using Windows, as pip doesn't support binary packages.
milliams 13y ago
Some of the mentioned problems with the traditional method are partially solved with `pip install requests --user` but I understand that the bigger problem/main reason for virtualenv isn't helped by this. However, I was very surprised that the author didn't mention venv (http://docs.python.org/3.3/library/venv.html http://docs.python.org/3.3/library/venv.html) at all since it is basically virtualenv but part of the Python standard library.
- tbatterii 13y ago
  i tried pyvenv last week. and it was pretty useless without pip + distribute. So virtualenv still wins IMO.
JulianWasTaken 13y ago
> Python actually has another, more primitive, package manager called easy_install, which is installed automatically when you install Python itself. It's actually not, it's part of setuptools/distribute, though some Python distributions (actually just brew that I know of) include distribute alongside Python. Also, while the quick skim of the rest of this looks mostly good, there's some unnecessary advice which complicates things. virtualenv is not necessary if the crux of the advice is "don't install packages globally" (which is fantastic advice, second probably to the more important advice "don't install stuff with sudo"). What beginners (and even some more experienced programmers) need to learn about is --user. You install packages with `pip install --user thing`, and they become per-user installed (be sure to add `~/.local/bin` to your `$PATH`). This is enough to not require sudo and for 99% of cases is actually sufficient. There is a rare case where you actually have packages that depend on different, conflicting versions of another dependency. This has happened ~1 time to me. Don't get me wrong, I like virtualenv for other reasons (relating to workflow and maintenance), but if we're teaching people about packaging, there's no particularly great reason to dedicate most of a tutorial on it.
- j4mie 13y ago
  Thanks, this is great advice. easy_install comes preinstalled on a Mac, which is where the confusion arose. Will update the article when I get a chance.
- euroclydon 13y ago
  > What beginners (and even some more experienced programmers) need to learn about is --user. You install packages with `pip install --user thing`, and they become per-user installed (be sure to add `~/.local/bin` to your `$PATH`). This is enough to not require sudo and for 99% of cases is actually sufficient. Is this relevant advice for packages needed by your web server account, e.g. www-data?
- pstuart 13y ago
  I recently learned the lesson about global packages. I thought it would be so nice to have all packages readily on hand, but now startup time is about 15 seconds of crawling the filesystem looking for files (over NFS). Go is looking more attractive day by day.....
HeyImAlex 13y ago
Want to make life even easier? Check out virtualenvwrapper. http://virtualenvwrapper.readthedocs.org/en/latest/ http://virtualenvwrapper.readthedocs.org/en/latest/ Lots of benefits, but trading 'cd path/to/my/project && source env/bin/activate' for 'workon project_env' (with autocomplete) is alone easily worth the five seconds it takes to check it out.
- almost 13y ago
  Definitely worth using as an experienced Python person. But you can skip a lot of confusion with new Python developers by waiting until later to introduce it :)
- j4mie 13y ago
  I find it's really useful for beginners to clearly understand what's actually happening under the covers before they start adding magical stuff on top. They can always add magic later, if they prefer that approach.
- pydanny 13y ago
  The only problem with virtualenvwrapper is setting it up can and does cause problems for beginning developers new to the shell. Otherwise it's really awesome. :-)
- elithrar 13y ago
  > Lots of benefits, but trading 'cd path/to/my/project && source env/bin/activate' for 'workon project_env' (with autocomplete) is alone easily worth the five seconds it takes to check it out. And/or use the virtualenvwrapper plug-in for oh-my-zsh. Automatically activates virtualenv when you cd to the working directory.
Pxtl 13y ago
VertualEnv seems less like a brilliant tool than it does a workaround for an architectural problem in Python and pip. That said, dependency hell is always tricky, and I've had to deal with some far uglier solutions in other platforms.
- tbatterii 13y ago
  you aren't implying that dependency hell is a problem unique to python/pip are you? dealing with dependencies is always step 2 for me when learning a new language and dependency hell seems like a universal problem. I could be wrong though.
  - tootie 13y ago
    Honestly, Maven is way easier, more powerful and works identically on Win, Linux or Mac. The key is that Java let's you set classpath as a command line arg and pythonpath is an environment variable.
    - tbatterii 13y ago
      IMO, buildout is more like maven for python. but again the tools solve this problem of "dependency hell" which to me is a problem all languages have, not just python
- metaphorm 13y ago
  it is a brilliant tool in that its an incredibly easy to use and lightweight solution to an otherwise very annoying problem. you might call it a workaround because its not the most elegant solution possible, but it still just works really well, so why complain?
nateleiby 13y ago
Thanks for the article! I recently spent some time going through the process of learning VirtualEnv / Pip; after looking at several tutorials, agreed that this is one of the better ones out there. A few other things I saw in other articles that you might like to clarify (though I understand there's a tradeoff with simplicity/clarity). (1) specifically state that `pip freeze` is how to create requirements file (as folks have said in comments already) (2) add "further reading" link on VirtualEnvWrapper, as it adds some convenience methods to ease use of VirtualEnv (3) the "yolk" package shows you what's currently installed; it can be helpful to `pip install yolk` then run `yolk -l` to view the different packages installed in each of your envs. (4) when installing a package, you can specify the version, e.g. `pip install mod==1.1`, whereas `pip install mod` will simply install the latest
rdtsc 13y ago
How does it play with OS packages? Do I package virtualenvs in an RPM or DEB?
- hoov 13y ago
  It's orthogonal. Create your base image, at deploy time, create a virtualenv and install your specific artifacts.
- deleted 13y ago
  [deleted]
mkoryak 13y ago
Can someone explain the historical reasons for this problem even existing in the first place? (the problem that virtual env solves)
- HeyImAlex 13y ago
  Did... did you read the article? Separation of environments for projects with different dependencies.
  - mkoryak 13y ago
    java does not have virtual envs, and i can have projects with different dependancies. clearly this is a python problem. I was asking why it exists.
- metaphorm 13y ago
  the problem is that different applications might require different versions of the same packages. if you install packages globally then you can only ever have one version available. this is a VERY BAD THING if you have any expectation of running more than application per machine.
- jamesjporter 13y ago
  It also comes in handy when I'm trying to install something finicky that has a million dependencies, each of which also might be finicky (e.g. numba) and I screw something up and just want to start over clean and not muck around uninstalling things. virtualenv makes this easy—I just make a new environment for experimenting and then delete it if I mess up and want to start over, leaving my system configuration clean and other environments intact.
- randuser 13y ago
  setuptools/distribute didn't implement `python setup.py uninstall`. Unix fragmentation of where the bin and lib directories should reside, i.e. /bin, /usr/bin, /usr/local/bin, ~/bin, ... Windows doesn't have symlinks and the different packaging tools have tried to implement the functionality in various different ways. Python doesn't add the path of the "main" executed file to the module lookup path. (edit: actually, I think this is wrong. I meant to say "Python module import lookup is complicated.")
Alex3917 13y ago
Thanks for this article, it was definitely needed. I've known that I should be using virtualenv for a while now, but actually trying to figure it out has been somewhat daunting until now.
toukon 13y ago
I'd like to know what the best practices with regards to security are for using pip, or installing packages in general. How do you verify package integrity? Do you simply pray that PyPI isn't compromised at the moment, or do you download your packages from Github instead, because the main repositories have more eyeballs on them? How do you do security updates with pip? I'm using apt-get at the moment which gives me security updates AFAIK, but my need is growing for more recent versions and certain packages that aren't accessible with apt.
- thruflo 13y ago
  One important note is to use pip>=1.3 (included in virtualenv>=1.9) as prior to this version, pip downloaded from pypi using http and was thus vulnerable to man in the middle attacks. You might also like to check out wheel, which allows you to compile signed binary distributions that you can install using pip.
didip 13y ago
Calling env/bin/python directly, as opposed to `source activate` is very handy for things like: * bash command line * cron * or daemon manager like supervisord
- thraxil 13y ago
  Yep. Years ago I settled on a convention of always installing a virtualenv into a 've' directory in the project so I can just set the shebang line on scripts (eg, django's manage.py) to "#!ve/bin/python". My Django project template sets all that up for me automatically. So now I just have muscle memory for typing "./manage.py ..." etc and I never have to activate virtualenvs, mess around with virtualenvwrapper type hacks or accidently run one project's script with a different project's virtualenv activated.
phren0logy 13y ago
Nice article, but after using leiningen (the clojure solution to a similar problem, based on maven), it's really hard to go back to something like this. I really, really wish there was an equivalent in python (really, every language I use).
Daiz 13y ago
Man, global system-wide installations that require admin rights by default? That's certainly something! Quite the stark comparison to Node.js and npm, where everything is installed locally into the current directory (under node_modules) by default, and "global" installation is actually a per-user installation. Tricking pip with virtualenv seems to get you pretty close to what you get by default with npm, albeit still somewhat more clunky. But to be fair, most other package managing solutions seem to pale in comparison to npm :-) Either way, nice article. Now if only most packages weren't still for Python 2... PyPI says it has 30099 packages total, but only around 2104 of them are for Python 3 (according to the number of entries on the "Python 3 Packages"-page[1]). [1] https://pypi.python.org/pypi?:action=browse&c=533&show=all https://pypi.python.org/pypi?:action=browse&c=533&sh...
- mixmastamyk 13y ago
  pip install --user XXX should get you what you want. Not default but not a huge burden either.
  - Daiz 13y ago
    ...Which would be equivalent to npm install -g XXX, whereas to replicate npm install XXX you'd need virtualenv. I don't think there even is an equivalent to pip install XXX without virtualenv with Node/npm (global system-wide installation for all users that requires admin rights).
- tomku 13y ago
  npm's default for -g is to install to Node's prefix, which is usually /usr or /usr/local. If you want it to install to your home directory, you can set the prefix to somewhere appropriate in your ~/.npmrc, which gives roughly the same behavior as pip's --user flag. Edit: perhaps you changed your .npmrc or set the option via npm and forgot about it? I just checked on a fresh user, and 'npm install -g' definitely tries to install to /usr, just like pip.
  - Daiz 13y ago
    I use Windows as my main OS, and by default npm -g installs packages to %AppData%, which is user-specific. I guess it's different on *nix, then.
rahduro 13y ago
Excellent tutorial, new to python, and I always hated when things just install without letting you know where it's going..and next time there is some upgrade all sorts of weird errors keep coming..Thanks a lot.
denzil_correa 13y ago
Python actually has another, more primitive, package manager called easy_install, which is installed automatically when you install Python itself. pip is vastly superior to easy_install for lots of reasons, and so should generally be used instead. You can use easy_install to install pip as follows: I found it quite ironic that the author says pip is "vastly superior" to easy_install and then proceeds to install pip using easy_install.
- kenny_r 13y ago
  It's similar to using Internet Explorer to download Chrome or Firefox.
- publicfig 13y ago
  Those are two non-related things, unless your only metric of superiority is that it comes pre-bundled.
hcarvalhoalves 13y ago
I would like to just give a difference advice regarding creating virtualenvs and installing dependencies: When you create the virtualenv, the current package you're working on doesn't get added to site-packages, so you're forced to be at the repository root to import the package. The best approach is to have a proper setup.py file so you can do `python setup.py develop`, which will link the package you're working on into the virtualenv site-packages. This way it acts as it's installed and you can import anyway you like. If you define your requirements on the setup.py (I think you should), you can even skip the `pip install -r requirements.txt` step. I've cooked up a package template that can help getting this working: https://github.com/hcarvalhoalves/python-package-template https://github.com/hcarvalhoalves/python-package-template
ak217 13y ago
Excellent, amazingly well-written guide. This should really be put on python.org as an "Intro to Python Package Management" or something.
pbreit 13y ago
For Mac, am I better off with Homebrew? http://docs.python-guide.org/en/latest/starting/install/osx/#install-osx http://docs.python-guide.org/en/latest/starting/install/osx/... https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python
- hosay123 13y ago
  Consider MacPorts over homebrew.. I'll withhold opinion on any system that turns the only sanctioned UNIX for site adminitrator's control ONLY directory in the system (since like the 80s) into a Git repository. MacPorts (and basically every other system of this kind) gets it right, unsurprisingly I'd never have believed a day would come where something like Homebrew would ever gain the traction it has.
- hcarvalhoalves 13y ago
  No, Homebrew solves a different problem (compiling and installing software in the global system). Think that virtualenv is way to package a complete Python environment together with the package you need or are working on.
- thruflo 13y ago
  No. Use homebrew for system packages and use pip to install python packages. It's much more flexible and doesn't rely on package managers keeping up with releases. In the real world, a typical pip requirements.txt file will have a mix of package names (which pip looks for and downloads from an index like pypi), git repo urls (eggs installed directly from a git server, eg from github) and bleeding edge track the latest changes -e git-repo#egg=eggname urls. That you can switch between these with ease is important, eg to switch to using your fork of some package rather than the last official release.
riobard 13y ago
Really hope someone could write a similar introduction to buildout (http://www.buildout.org/ http://www.buildout.org/)
- e12e 13y ago
  You might enjoy: http://jacobian.org/writing/django-apps-with-buildout/ http://jacobian.org/writing/django-apps-with-buildout/ Which covers a lot of the same ground, but with buildout.
vram22 13y ago
Interesting article.
yew 13y ago
Speaking as someone relatively new to Python (coming from an embedded development background, mostly with C/C++): What's the standard way of distributing client-side programs/libraries? If you only have one script you can just put it in /usr/local/bin/ but otherwise you have to mess with the system site-packages or modify sys.path before importing, right? I've seen a surprisingly large number of distros that didn't check /usr/ and /usr/local/ for packages. Do you just hand the user an automatic virtualenv script? (Outside of using one of the binary builders out there, obviously.)

241 points by j4mie 13y ago

Flawless explanation. As a somewhat beginner to python i must give my thanks !

a good introduction - I would like to hear more about deployment with virtualenv though - is it expected that you just document any packages with requirements.txt and then you would create the virtualenv in the deployment target and set everything up again? Or can you "package" a virtualenv for deployment?

HeyImAlex 13y ago
Just generate your requirements file from your env locally (pip freeze > requirements.txt), deploy all of your files (env folder excluded) to your sever however you want and then run 'env/bin/pip install -r requirements.txt' on your server.
- turtle4 13y ago
  If I use pip freeze > requirements.txt, it lists the packages I have installed using pip, but it doesn't list anything for the python version itself. How do I make sure the right python version gets captured if I don't check in the /env/ folder?
deleted 13y ago
[deleted]
fernly 13y ago
Deployment/distribution can be handled by bundling the app. For Python 2.x, there's PyInstaller[0], Py2App[1], Py2Exe[2] all of which do much the same thing: make a single binary out of a python app and all its dependencies including the interpreter. Then you distribute that and don't worry about what the user has or hasn't got. [0]http://www.pyinstaller.org/ http://www.pyinstaller.org/ [1]https://pypi.python.org/pypi/py2app/ https://pypi.python.org/pypi/py2app/ [2]https://pypi.python.org/pypi/py2exe/0.6.9 https://pypi.python.org/pypi/py2exe/0.6.9

tocomment 13y ago

He mentions not checking the env directory into git. Why not? In general what are the best practices for using virtualenv with version control?

afandian 13y ago
What if you want to perform a security update for one of the libraries on the server? Also, IIRC the environment will contain a symlimk to the python executable and companion files. That symlink will change depending on the environment.
kingrolo 13y ago
Keep your requirements file checked in, but not the virtualenv. The built env should be seen as disposable, and is both location (ie, path on disk) and platform (for libs which are not pure python) specific . I keep my virtualenvs in ~/.virtualenvs/, away from the project.
cinquemb 13y ago
You could have issues with other people not running the same version of python, also you might have different site packages if you are working on an experimental branch that will be pushed or merged later.
almost 13y ago
Virtual envs don't relocate well, they tend to be very specific to machine and even install location. Plus you can recreate them from your requirements.txt file so there's no need. Just add the env directory to your .gitignore
arnarbi 13y ago
I find it best to keep virtual envs completely away from the project (I use http://virtualenvwrapper.readthedocs.org/en/latest/ http://virtualenvwrapper.readthedocs.org/en/latest/ which puts them by default in ~/.virtualenvs). A virtualenv is completely machine-specific. If your project is a package itself (i.e. it has a setup.py file), then use that file to specify dependencies. On a new machine I check out a copy, create a virtual env and activate it. Then in the local copy I run "pip install -e .". This installs all the requirements from setup.py in the virtualenv, and links the local copy of my project to it as well. Now your package is available in the virtual env, but fully editable. If your python project is not a package, you can install its dependencies in a virtual env with pip. Then run "pip freeze" to generate a list of all installed packages. Save that to a text file in your repository, e.g. ``requirements.txt``. On a different machine, or a fresh venv, you can then do "pip install -r requirements.txt" to set everything up in one go.

anonozdotcom 13y ago

My mentor did not even give such a good introduction. Kudos to the author, now I have a bright AHHA lightbulb in my head! :D

deleted 13y ago
[deleted]
suyash 13y ago
That is why they say to take multiple advice from multiple sources.

almost 13y ago

If you're in Brighton in the UK and you like this then maybe you'll be interested in the one day workshop run by Jamie (author of the blog post) and myself. This post was actually based on some of the material written by Jamie for the course. Next one is happening next Thursday and there are still a couple of tickets: http://dabapps.com/services/training/python-for-programmers http://dabapps.com/services/training/python-for-programmers

wmt 13y ago

"pip is vastly superior toeasy_install for lots of reasons, and so should generally be used instead." Unless you are using Windows, as pip doesn't support binary packages.

milliams 13y ago

Some of the mentioned problems with the traditional method are partially solved with `pip install requests --user` but I understand that the bigger problem/main reason for virtualenv isn't helped by this. However, I was very surprised that the author didn't mention venv (http://docs.python.org/3.3/library/venv.html http://docs.python.org/3.3/library/venv.html) at all since it is basically virtualenv but part of the Python standard library.

tbatterii 13y ago
i tried pyvenv last week. and it was pretty useless without pip + distribute. So virtualenv still wins IMO.

JulianWasTaken 13y ago

> Python actually has another, more primitive, package manager called easy_install, which is installed automatically when you install Python itself. It's actually not, it's part of setuptools/distribute, though some Python distributions (actually just brew that I know of) include distribute alongside Python. Also, while the quick skim of the rest of this looks mostly good, there's some unnecessary advice which complicates things. virtualenv is not necessary if the crux of the advice is "don't install packages globally" (which is fantastic advice, second probably to the more important advice "don't install stuff with sudo"). What beginners (and even some more experienced programmers) need to learn about is --user. You install packages with `pip install --user thing`, and they become per-user installed (be sure to add `~/.local/bin` to your `$PATH`). This is enough to not require sudo and for 99% of cases is actually sufficient. There is a rare case where you actually have packages that depend on different, conflicting versions of another dependency. This has happened ~1 time to me. Don't get me wrong, I like virtualenv for other reasons (relating to workflow and maintenance), but if we're teaching people about packaging, there's no particularly great reason to dedicate most of a tutorial on it.

j4mie 13y ago
Thanks, this is great advice. easy_install comes preinstalled on a Mac, which is where the confusion arose. Will update the article when I get a chance.
euroclydon 13y ago
> What beginners (and even some more experienced programmers) need to learn about is --user. You install packages with `pip install --user thing`, and they become per-user installed (be sure to add `~/.local/bin` to your `$PATH`). This is enough to not require sudo and for 99% of cases is actually sufficient. Is this relevant advice for packages needed by your web server account, e.g. www-data?
pstuart 13y ago
I recently learned the lesson about global packages. I thought it would be so nice to have all packages readily on hand, but now startup time is about 15 seconds of crawling the filesystem looking for files (over NFS). Go is looking more attractive day by day.....

HeyImAlex 13y ago

Want to make life even easier? Check out virtualenvwrapper. http://virtualenvwrapper.readthedocs.org/en/latest/ http://virtualenvwrapper.readthedocs.org/en/latest/ Lots of benefits, but trading 'cd path/to/my/project && source env/bin/activate' for 'workon project_env' (with autocomplete) is alone easily worth the five seconds it takes to check it out.

almost 13y ago
Definitely worth using as an experienced Python person. But you can skip a lot of confusion with new Python developers by waiting until later to introduce it :)
j4mie 13y ago
I find it's really useful for beginners to clearly understand what's actually happening under the covers before they start adding magical stuff on top. They can always add magic later, if they prefer that approach.
pydanny 13y ago
The only problem with virtualenvwrapper is setting it up can and does cause problems for beginning developers new to the shell. Otherwise it's really awesome. :-)
elithrar 13y ago
> Lots of benefits, but trading 'cd path/to/my/project && source env/bin/activate' for 'workon project_env' (with autocomplete) is alone easily worth the five seconds it takes to check it out. And/or use the virtualenvwrapper plug-in for oh-my-zsh. Automatically activates virtualenv when you cd to the working directory.

Pxtl 13y ago

VertualEnv seems less like a brilliant tool than it does a workaround for an architectural problem in Python and pip. That said, dependency hell is always tricky, and I've had to deal with some far uglier solutions in other platforms.

tbatterii 13y ago
you aren't implying that dependency hell is a problem unique to python/pip are you? dealing with dependencies is always step 2 for me when learning a new language and dependency hell seems like a universal problem. I could be wrong though.
- tootie 13y ago
  Honestly, Maven is way easier, more powerful and works identically on Win, Linux or Mac. The key is that Java let's you set classpath as a command line arg and pythonpath is an environment variable.
  - tbatterii 13y ago
    IMO, buildout is more like maven for python. but again the tools solve this problem of "dependency hell" which to me is a problem all languages have, not just python
metaphorm 13y ago
it is a brilliant tool in that its an incredibly easy to use and lightweight solution to an otherwise very annoying problem. you might call it a workaround because its not the most elegant solution possible, but it still just works really well, so why complain?

nateleiby 13y ago

Thanks for the article! I recently spent some time going through the process of learning VirtualEnv / Pip; after looking at several tutorials, agreed that this is one of the better ones out there. A few other things I saw in other articles that you might like to clarify (though I understand there's a tradeoff with simplicity/clarity). (1) specifically state that `pip freeze` is how to create requirements file (as folks have said in comments already) (2) add "further reading" link on VirtualEnvWrapper, as it adds some convenience methods to ease use of VirtualEnv (3) the "yolk" package shows you what's currently installed; it can be helpful to `pip install yolk` then run `yolk -l` to view the different packages installed in each of your envs. (4) when installing a package, you can specify the version, e.g. `pip install mod==1.1`, whereas `pip install mod` will simply install the latest

rdtsc 13y ago

How does it play with OS packages? Do I package virtualenvs in an RPM or DEB?

hoov 13y ago
It's orthogonal. Create your base image, at deploy time, create a virtualenv and install your specific artifacts.
deleted 13y ago
[deleted]

mkoryak 13y ago

Can someone explain the historical reasons for this problem even existing in the first place? (the problem that virtual env solves)

HeyImAlex 13y ago
Did... did you read the article? Separation of environments for projects with different dependencies.
- mkoryak 13y ago
  java does not have virtual envs, and i can have projects with different dependancies. clearly this is a python problem. I was asking why it exists.
metaphorm 13y ago
the problem is that different applications might require different versions of the same packages. if you install packages globally then you can only ever have one version available. this is a VERY BAD THING if you have any expectation of running more than application per machine.
jamesjporter 13y ago
It also comes in handy when I'm trying to install something finicky that has a million dependencies, each of which also might be finicky (e.g. numba) and I screw something up and just want to start over clean and not muck around uninstalling things. virtualenv makes this easy—I just make a new environment for experimenting and then delete it if I mess up and want to start over, leaving my system configuration clean and other environments intact.
randuser 13y ago
setuptools/distribute didn't implement `python setup.py uninstall`. Unix fragmentation of where the bin and lib directories should reside, i.e. /bin, /usr/bin, /usr/local/bin, ~/bin, ... Windows doesn't have symlinks and the different packaging tools have tried to implement the functionality in various different ways. Python doesn't add the path of the "main" executed file to the module lookup path. (edit: actually, I think this is wrong. I meant to say "Python module import lookup is complicated.")

Alex3917 13y ago

Thanks for this article, it was definitely needed. I've known that I should be using virtualenv for a while now, but actually trying to figure it out has been somewhat daunting until now.

toukon 13y ago

I'd like to know what the best practices with regards to security are for using pip, or installing packages in general. How do you verify package integrity? Do you simply pray that PyPI isn't compromised at the moment, or do you download your packages from Github instead, because the main repositories have more eyeballs on them? How do you do security updates with pip? I'm using apt-get at the moment which gives me security updates AFAIK, but my need is growing for more recent versions and certain packages that aren't accessible with apt.

thruflo 13y ago
One important note is to use pip>=1.3 (included in virtualenv>=1.9) as prior to this version, pip downloaded from pypi using http and was thus vulnerable to man in the middle attacks. You might also like to check out wheel, which allows you to compile signed binary distributions that you can install using pip.

didip 13y ago

Calling env/bin/python directly, as opposed to `source activate` is very handy for things like: * bash command line * cron * or daemon manager like supervisord

thraxil 13y ago
Yep. Years ago I settled on a convention of always installing a virtualenv into a 've' directory in the project so I can just set the shebang line on scripts (eg, django's manage.py) to "#!ve/bin/python". My Django project template sets all that up for me automatically. So now I just have muscle memory for typing "./manage.py ..." etc and I never have to activate virtualenvs, mess around with virtualenvwrapper type hacks or accidently run one project's script with a different project's virtualenv activated.

phren0logy 13y ago

Nice article, but after using leiningen (the clojure solution to a similar problem, based on maven), it's really hard to go back to something like this. I really, really wish there was an equivalent in python (really, every language I use).

Daiz 13y ago

Man, global system-wide installations that require admin rights by default? That's certainly something! Quite the stark comparison to Node.js and npm, where everything is installed locally into the current directory (under node_modules) by default, and "global" installation is actually a per-user installation. Tricking pip with virtualenv seems to get you pretty close to what you get by default with npm, albeit still somewhat more clunky. But to be fair, most other package managing solutions seem to pale in comparison to npm :-) Either way, nice article. Now if only most packages weren't still for Python 2... PyPI says it has 30099 packages total, but only around 2104 of them are for Python 3 (according to the number of entries on the "Python 3 Packages"-page[1]). [1] https://pypi.python.org/pypi?:action=browse&c=533&show=all https://pypi.python.org/pypi?:action=browse&c=533&sh...

mixmastamyk 13y ago
pip install --user XXX should get you what you want. Not default but not a huge burden either.
- Daiz 13y ago
  ...Which would be equivalent to npm install -g XXX, whereas to replicate npm install XXX you'd need virtualenv. I don't think there even is an equivalent to pip install XXX without virtualenv with Node/npm (global system-wide installation for all users that requires admin rights).
tomku 13y ago
npm's default for -g is to install to Node's prefix, which is usually /usr or /usr/local. If you want it to install to your home directory, you can set the prefix to somewhere appropriate in your ~/.npmrc, which gives roughly the same behavior as pip's --user flag. Edit: perhaps you changed your .npmrc or set the option via npm and forgot about it? I just checked on a fresh user, and 'npm install -g' definitely tries to install to /usr, just like pip.
- Daiz 13y ago
  I use Windows as my main OS, and by default npm -g installs packages to %AppData%, which is user-specific. I guess it's different on *nix, then.

rahduro 13y ago

Excellent tutorial, new to python, and I always hated when things just install without letting you know where it's going..and next time there is some upgrade all sorts of weird errors keep coming..Thanks a lot.

denzil_correa 13y ago

Python actually has another, more primitive, package manager called easy_install, which is installed automatically when you install Python itself. pip is vastly superior to easy_install for lots of reasons, and so should generally be used instead. You can use easy_install to install pip as follows: I found it quite ironic that the author says pip is "vastly superior" to easy_install and then proceeds to install pip using easy_install.

kenny_r 13y ago
It's similar to using Internet Explorer to download Chrome or Firefox.
publicfig 13y ago
Those are two non-related things, unless your only metric of superiority is that it comes pre-bundled.

hcarvalhoalves 13y ago

I would like to just give a difference advice regarding creating virtualenvs and installing dependencies: When you create the virtualenv, the current package you're working on doesn't get added to site-packages, so you're forced to be at the repository root to import the package. The best approach is to have a proper setup.py file so you can do `python setup.py develop`, which will link the package you're working on into the virtualenv site-packages. This way it acts as it's installed and you can import anyway you like. If you define your requirements on the setup.py (I think you should), you can even skip the `pip install -r requirements.txt` step. I've cooked up a package template that can help getting this working: https://github.com/hcarvalhoalves/python-package-template https://github.com/hcarvalhoalves/python-package-template

ak217 13y ago

Excellent, amazingly well-written guide. This should really be put on python.org as an "Intro to Python Package Management" or something.

pbreit 13y ago

For Mac, am I better off with Homebrew? http://docs.python-guide.org/en/latest/starting/install/osx/#install-osx http://docs.python-guide.org/en/latest/starting/install/osx/... https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python

hosay123 13y ago
Consider MacPorts over homebrew.. I'll withhold opinion on any system that turns the only sanctioned UNIX for site adminitrator's control ONLY directory in the system (since like the 80s) into a Git repository. MacPorts (and basically every other system of this kind) gets it right, unsurprisingly I'd never have believed a day would come where something like Homebrew would ever gain the traction it has.
hcarvalhoalves 13y ago
No, Homebrew solves a different problem (compiling and installing software in the global system). Think that virtualenv is way to package a complete Python environment together with the package you need or are working on.
thruflo 13y ago
No. Use homebrew for system packages and use pip to install python packages. It's much more flexible and doesn't rely on package managers keeping up with releases. In the real world, a typical pip requirements.txt file will have a mix of package names (which pip looks for and downloads from an index like pypi), git repo urls (eggs installed directly from a git server, eg from github) and bleeding edge track the latest changes -e git-repo#egg=eggname urls. That you can switch between these with ease is important, eg to switch to using your fork of some package rather than the last official release.

riobard 13y ago

Really hope someone could write a similar introduction to buildout (http://www.buildout.org/ http://www.buildout.org/)

e12e 13y ago
You might enjoy: http://jacobian.org/writing/django-apps-with-buildout/ http://jacobian.org/writing/django-apps-with-buildout/ Which covers a lot of the same ground, but with buildout.

vram22 13y ago

Interesting article.

yew 13y ago

Speaking as someone relatively new to Python (coming from an embedded development background, mostly with C/C++): What's the standard way of distributing client-side programs/libraries? If you only have one script you can just put it in /usr/local/bin/ but otherwise you have to mess with the system site-packages or modify sys.path before importing, right? I've seen a surprisingly large number of distros that didn't check /usr/ and /usr/local/ for packages. Do you just hand the user an automatic virtualenv script? (Outside of using one of the binary builders out there, obviously.)