4 ms·

Spent some time this week to move deployment of venvs to use wheels (we install numpy and matplotlib). Now, instead of close to 10 minutes deployment time, it's

by sashk 13y ago

Spent some time this week to move deployment of venvs to use wheels (we install numpy and matplotlib). Now, instead of close to 10 minutes deployment time, it's done in about 30 seconds. I can live with the fact, that I'll need to make changes to the deployment process now and then, while wheel traveling towards 1.0, but I'm happy to go that route. And you should to.

gtaylor 13y ago
Isn't there a way to make a relocatable virtualenv? I'd be inclined to just do that, compress it up and distribute the entire virtualenv (our deploy environment is homogenous to the extreme).
- ot 13y ago
  A few years ago I wrote a tool to do exactly this [1], which also works for any executable/library, not just Python code. It works by compiling the code with a unique virtual prefix that is something like /tmp/boxes/4031e76a-6bff-11e3-a3b0-002590a9f2cc so that it can be symlinked to any directory in the filesystem. The tool also generates a script that sets environment variables so that includes, libraries, executables, man pages, etc... are all found in the path. It also allows to install several versions of the same package and instantly switching from a version to another, which is very useful during development. Unfortunately I didn't have more time to work on it so it is basically unmaintained, but every now and then I still use it, mainly when I need to install software on machines where I don't have administrative privileges, and it works great. I wish I implemented some kind of dependency handling mechanism, it would have made it much more useful. [1] https://github.com/ot/bpt https://github.com/ot/bpt
- travisoliphant 13y ago
  Conda (http://conda.pydata.org http://conda.pydata.org) is a more general packaging format than wheels and we use it to distribute the Scientific Python Stack (including complex C-dependencies) with ease. It is more general than wheels and solves this fundamental problem now with all the benefits without waiting for something in the future. See these blog-posts for more information: http://www.continuum.io/blog/conda_packaging http://www.continuum.io/blog/conda_packaging and http://technicaldiscovery.blogpost.com http://technicaldiscovery.blogpost.com In particular look at the `conda package` command for an easy way to bundle up any non-conda packages across a homogeneous environment.
  - dcuthbertson 13y ago
    FYI: I think your last link should be "http://technicaldiscovery.blogspot.com/" http://technicaldiscovery.blogspot.com/". There doesn't seem to be an active site at blogpost.com.
    - travisoliphant 13y ago
      Yes, thank you!. You have the correct link.
- donaldstufft 13y ago
  There is a way to make a virtualenv relocatable but it has some issues where it doesn't always work very well. I'm not sure offhand what those issues are though.
  - count 13y ago
    Isn't that basically docker?
    - mh- 13y ago
      no. oversimplifying here, but.. virtualenvs are just overrides to environment variables that change where things should look to find the python interpreter, libraries, modules, et al. docker is much more involved, using LXC to force process isolation, an effective chroot, networking restrictions, etc. many (most?) folks' current usage of virtualenv would not be portable to docker without substantive work to accomodate the restrictions that docker imposes. (note: this isn't a critique of docker; those things are features. they're just orthogonal to this)

by sashk 13y ago

gtaylor 13y ago

Isn't there a way to make a relocatable virtualenv? I'd be inclined to just do that, compress it up and distribute the entire virtualenv (our deploy environment is homogenous to the extreme).

ot 13y ago
A few years ago I wrote a tool to do exactly this [1], which also works for any executable/library, not just Python code. It works by compiling the code with a unique virtual prefix that is something like /tmp/boxes/4031e76a-6bff-11e3-a3b0-002590a9f2cc so that it can be symlinked to any directory in the filesystem. The tool also generates a script that sets environment variables so that includes, libraries, executables, man pages, etc... are all found in the path. It also allows to install several versions of the same package and instantly switching from a version to another, which is very useful during development. Unfortunately I didn't have more time to work on it so it is basically unmaintained, but every now and then I still use it, mainly when I need to install software on machines where I don't have administrative privileges, and it works great. I wish I implemented some kind of dependency handling mechanism, it would have made it much more useful. [1] https://github.com/ot/bpt https://github.com/ot/bpt
travisoliphant 13y ago
Conda (http://conda.pydata.org http://conda.pydata.org) is a more general packaging format than wheels and we use it to distribute the Scientific Python Stack (including complex C-dependencies) with ease. It is more general than wheels and solves this fundamental problem now with all the benefits without waiting for something in the future. See these blog-posts for more information: http://www.continuum.io/blog/conda_packaging http://www.continuum.io/blog/conda_packaging and http://technicaldiscovery.blogpost.com http://technicaldiscovery.blogpost.com In particular look at the `conda package` command for an easy way to bundle up any non-conda packages across a homogeneous environment.
- dcuthbertson 13y ago
  FYI: I think your last link should be "http://technicaldiscovery.blogspot.com/" http://technicaldiscovery.blogspot.com/". There doesn't seem to be an active site at blogpost.com.
  - travisoliphant 13y ago
    Yes, thank you!. You have the correct link.
donaldstufft 13y ago
There is a way to make a virtualenv relocatable but it has some issues where it doesn't always work very well. I'm not sure offhand what those issues are though.
- count 13y ago
  Isn't that basically docker?
  - mh- 13y ago
    no. oversimplifying here, but.. virtualenvs are just overrides to environment variables that change where things should look to find the python interpreter, libraries, modules, et al. docker is much more involved, using LXC to force process isolation, an effective chroot, networking restrictions, etc. many (most?) folks' current usage of virtualenv would not be portable to docker without substantive work to accomodate the restrictions that docker imposes. (note: this isn't a critique of docker; those things are features. they're just orthogonal to this)