16 ms·

Python on Wheels

151 points by donaldstufft 13y ago

deleted 13y ago
[deleted]
sashk 13y ago
Spent some time this week to move deployment of venvs to use wheels (we install numpy and matplotlib). Now, instead of close to 10 minutes deployment time, it's done in about 30 seconds. I can live with the fact, that I'll need to make changes to the deployment process now and then, while wheel traveling towards 1.0, but I'm happy to go that route. And you should to.
- gtaylor 13y ago
  Isn't there a way to make a relocatable virtualenv? I'd be inclined to just do that, compress it up and distribute the entire virtualenv (our deploy environment is homogenous to the extreme).
  - ot 13y ago
    A few years ago I wrote a tool to do exactly this [1], which also works for any executable/library, not just Python code. It works by compiling the code with a unique virtual prefix that is something like /tmp/boxes/4031e76a-6bff-11e3-a3b0-002590a9f2cc so that it can be symlinked to any directory in the filesystem. The tool also generates a script that sets environment variables so that includes, libraries, executables, man pages, etc... are all found in the path. It also allows to install several versions of the same package and instantly switching from a version to another, which is very useful during development. Unfortunately I didn't have more time to work on it so it is basically unmaintained, but every now and then I still use it, mainly when I need to install software on machines where I don't have administrative privileges, and it works great. I wish I implemented some kind of dependency handling mechanism, it would have made it much more useful. [1] https://github.com/ot/bpt https://github.com/ot/bpt
  - travisoliphant 13y ago
    Conda (http://conda.pydata.org http://conda.pydata.org) is a more general packaging format than wheels and we use it to distribute the Scientific Python Stack (including complex C-dependencies) with ease. It is more general than wheels and solves this fundamental problem now with all the benefits without waiting for something in the future. See these blog-posts for more information: http://www.continuum.io/blog/conda_packaging http://www.continuum.io/blog/conda_packaging and http://technicaldiscovery.blogpost.com http://technicaldiscovery.blogpost.com In particular look at the `conda package` command for an easy way to bundle up any non-conda packages across a homogeneous environment.
    - dcuthbertson 13y ago
      FYI: I think your last link should be "http://technicaldiscovery.blogspot.com/" http://technicaldiscovery.blogspot.com/". There doesn't seem to be an active site at blogpost.com.
      - travisoliphant 13y ago
        Yes, thank you!. You have the correct link.
  - donaldstufft 13y ago
    There is a way to make a virtualenv relocatable but it has some issues where it doesn't always work very well. I'm not sure offhand what those issues are though.
    - count 13y ago
      Isn't that basically docker?
      - mh- 13y ago
        no. oversimplifying here, but.. virtualenvs are just overrides to environment variables that change where things should look to find the python interpreter, libraries, modules, et al. docker is much more involved, using LXC to force process isolation, an effective chroot, networking restrictions, etc. many (most?) folks' current usage of virtualenv would not be portable to docker without substantive work to accomodate the restrictions that docker imposes. (note: this isn't a critique of docker; those things are features. they're just orthogonal to this)
deleted 13y ago
[deleted]
raptium 13y ago
I thought it was another web framework ...
- r0muald 13y ago
  Just like Ruby on Rails :)
mariocesar 13y ago
I use wheels to speed up the tests for sorl-thumbnail https://github.com/mariocesar/sorl-thumbnail/blob/master/.travis.yml#L26 https://github.com/mariocesar/sorl-thumbnail/blob/master/.tr... Initially I was into fully use wheel to install the environment but I found that some packages didn't work, now I just use it for just some critical packages and reduce the 12 minutes test to 3 minutes Will love to see wheels getting more attention, really feels like a great direction for python packaging.
- jpdlla 13y ago
  Love sorl-thumbnail, thanks for all the great work!
- travisoliphant 13y ago
  A lot of people in the PyData ecosystem using NumPy and Pandas are using conda and conda packcages (http://conda.pydata.org http://conda.pydata.org). Conda packages are easy to build and many exist at repo.continuum.io and on channels at http://binstar.org http://binstar.org
girvo 13y ago
For a language that prides itself on simplicity, this seems really painful...
- pak 13y ago
  Yes, it is. Coming to this mess from the joy of bundler/gems, npm, or hell, even CPAN will ruin your afternoon. It is just an unfortunate circumstance of the community not efficiently converging on well designed standards quickly enough, and so, the rocky soil has sprouted strange twisty crops. There are articles on how the setuptools/distutils frankenhack pretty much set off the whole trainwreck that you see today, e.g.: http://blog.startifact.com/posts/older/a-history-of-python-packaging.html http://blog.startifact.com/posts/older/a-history-of-python-p... Armin (the OP) has also written about this sad history previously: http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
  - mixmastamyk 13y ago
    Would be nice if there were one packaging tool all those languages were using instead of each one reinventing it.
    - rmc 13y ago
      Something that integrated into your operating system distribution perhaps....
      - mh- 13y ago
        aptly put. you might be onto something.
        mwexler 13y ago
        Groan. Well put, but still, groan. 8-)
        mitchty 13y ago
pippy 13y ago
I've had the exact same experience as the author of this post. Eggs are great if you just want to install a python library. Horrible if you want to modify the library or distribute a program. I personally use a mixture of setuptools and distutils2, and it's a confusing mix.
shadowmint 13y ago
Wow, great article. I've heard of wheels for months, but I've never really understood why I should remotely care about them before (it always seemed to me they solved a problem that didn't exist; namely, binary distributions for pure python packages, since you can't support c-plugin packages cleanly with wheel anyhow...). Turns out the answer is: 1) Yes, you can kind of support packages with c-plugins, if you're specific and careful. 2) They make server deployments significantly faster. That's actually pretty neat. ...but damn, they still look like a massive pain to work with.
- brabram 13y ago
  I confirm about the pain. They should be transparent and you shouldn't need to care about them but that's absolutely not the case. Also, it's the responsibility of the pypi maintener to push wheels on it, but none do this, so you end up building your own wheels while this should have already been done on pypi and it's stupid. And one last thing: https://twitter.com/mitsuhiko/status/426700148409135104 https://twitter.com/mitsuhiko/status/426700148409135104 "It turns out, Python binary wheels on Python 2 are rejected by PyPI, I suppose because of the lack of UCS tag in the filename." Apart from all of this pain, I'm very happy to be able to use them. I'm really waiting for someone to build a "wheel on demand" website that would be a proxy to pypi. Or to turn pypi into a wheel farm (I don't know how realist this idea is).
  - donaldstufft 13y ago
    <- PyPI Administrator. I have plans for either turning PyPI into a build farm, or making a secondary service that acts as a build farm for PyPI. We've just been more focused on cleaning up other issues.
- themartorana 13y ago
  After being bummed by the removal of 'bundle' in pip and writing an article about how we moved to using wheel for faster deployments[1], we moved to using Packer for server builds for our cloud infrastructure. Now we can backup to using a simple 'requirements.txt' file and not worrying about install and compile speeds. Removing 'bundle' was painful for us, and I even contributed a bunch of code to 'pip2pi' [2] to make it easy to set up an S3 bucket as a "local" pip mirror for our EC2 infrastructure, but it was all bandaid after bandaid. So far, Packer has felt much more elegant (despite the effort it took to put it in to our workflow) than wheel, pip2pi, or any other solution. [1] http://tech.flyclops.com/replacing-pip-bundle-374 http://tech.flyclops.com/replacing-pip-bundle-374 [2] https://github.com/wolever/pip2pi https://github.com/wolever/pip2pi (Edited for grammar.)
e12e 13y ago
A great introduction to wheels. I'm still not sure if wheels really is a good idea though. For simple use-cases traditonal pip seems fine, for more complex use-cases buildout seems better? Having tried to install numpy and pygame in a virtualenv I do see a need for something better -- but I'm not convinced wheels will solve the problem of complex c-library dependencies...?
_jb 13y ago
Google cache link: http://webcache.googleusercontent.com/search?q=cache:0rftZaFa274J:lucumr.pocoo.org/2014/1/27/python-on-wheels/+&cd=1&hl=en&ct=clnk&gl=ca http://webcache.googleusercontent.com/search?q=cache:0rftZaF...
beggi 13y ago
Page is currently down, so here: http://webcache.googleusercontent.com/search?q=cache:http://lucumr.pocoo.org/2014/1/27/python-on-wheels/ http://webcache.googleusercontent.com/search?q=cache:http://...
amouat 13y ago
On a slight tangent; does anyone else think that Docker is probably a better solution than virtualenv in most cases? I never really got on withe virtualenv.
- dagw 13y ago
  The subset of problems that virtualenv solves that are also solvable by Docker is pretty tiny. But sure if you have problem that can easily be solved by Docker, then Docker is probably a better solution.
  - amouat 13y ago
    Really? I was thinking you just set up a new Docker env for each python project, then you don't need to worry about conflicting with libs from other projects. What am I missing?
    - dagw 13y ago
      Docker only runs on a small subset of the platforms python runs on is the most obvious one. Can docker easily access the host filesystem or do you have to copy all your data into each docker env? Can I access my GPU for doing CUDA/OpenGL/OpenCL things from withing docker? Does docker play nicely with GUI apps? Calling into a running docker env from an external program (when you are using an app that has python as scripting or plug-in language) doesn't work as far as I know. The overhead of setting up, tearing down an switching between several Docker envs seems to higher than with tools like virtualenvwrapper or conda. Now there may be workarounds for these (and all the other) problems but on the surface they don't seem easier than virtualenv.
      - amouat 13y ago
        Ok, thanks for the reply. Good questions. I was mainly thinking in the context of developing webapps. I'm interested in this enough that I intend to investigate and answer all your questions, perhaps in a blog post. I don't know enough right now to answer with certainty. However: - it will currently only work well if you are developing on Linux. MacOS isn't bad but requires vagrant/virtualbox. - I suspect switching between Docker envs may be faster, easier and cleaner than virtualenv which is why I suggested it. - There is definitely support for sharing data.
overgard 13y ago
Ugh. I'm so sick of the open source communities' disdain for binaries. There's no reason we should be recompiling shit everywhere. It's a waste of time, headspace, and processing power.
- Pxtl 13y ago
  When you're trying to support multiple architectures and OSes and your ABI took a decade or two to settle down, and all your tools are open-source so you have the source anyways, I can see the instinct to just say "I'll just compile it against the target machine" even if it's absurdly wasteful.
  - hackerboos 13y ago
    Popular binaries can be uploaded falling back to building from source.
DrJ 13y ago
what about pex?
travisoliphant 13y ago
This is a nice post, and the Python packaging community is making strides in the right direction. However, it's not there yet. Armin could bring himself to say this in this post: "It's there, it kinda works, and it's probably worth your time." Conda is not perfect (it needs more people building conda binaries as well as a few pull requests to add https improvements) --- but I will confidently say "it is there, it does work now, and it is worth your time" This is especially true with a package repository like exists at http://repo.continuum.io http://repo.continuum.io and the many personal repositories at http://binstar.org http://binstar.org. People that have used conda for deployment (especially of complex C-dependencies in the NumPy and PyData stack) are saving time and effort today.
asmeurer 13y ago
"What the command is supposed to do is to collect all the dependencies and the convert them into wheels if necessary..." How does it convert the non-wheel packages into wheels?

151 points by donaldstufft 13y ago

deleted 13y ago

[deleted]

sashk 13y ago

Spent some time this week to move deployment of venvs to use wheels (we install numpy and matplotlib). Now, instead of close to 10 minutes deployment time, it's done in about 30 seconds. I can live with the fact, that I'll need to make changes to the deployment process now and then, while wheel traveling towards 1.0, but I'm happy to go that route. And you should to.

gtaylor 13y ago
Isn't there a way to make a relocatable virtualenv? I'd be inclined to just do that, compress it up and distribute the entire virtualenv (our deploy environment is homogenous to the extreme).
- ot 13y ago
  A few years ago I wrote a tool to do exactly this [1], which also works for any executable/library, not just Python code. It works by compiling the code with a unique virtual prefix that is something like /tmp/boxes/4031e76a-6bff-11e3-a3b0-002590a9f2cc so that it can be symlinked to any directory in the filesystem. The tool also generates a script that sets environment variables so that includes, libraries, executables, man pages, etc... are all found in the path. It also allows to install several versions of the same package and instantly switching from a version to another, which is very useful during development. Unfortunately I didn't have more time to work on it so it is basically unmaintained, but every now and then I still use it, mainly when I need to install software on machines where I don't have administrative privileges, and it works great. I wish I implemented some kind of dependency handling mechanism, it would have made it much more useful. [1] https://github.com/ot/bpt https://github.com/ot/bpt
- travisoliphant 13y ago
  Conda (http://conda.pydata.org http://conda.pydata.org) is a more general packaging format than wheels and we use it to distribute the Scientific Python Stack (including complex C-dependencies) with ease. It is more general than wheels and solves this fundamental problem now with all the benefits without waiting for something in the future. See these blog-posts for more information: http://www.continuum.io/blog/conda_packaging http://www.continuum.io/blog/conda_packaging and http://technicaldiscovery.blogpost.com http://technicaldiscovery.blogpost.com In particular look at the `conda package` command for an easy way to bundle up any non-conda packages across a homogeneous environment.
  - dcuthbertson 13y ago
    FYI: I think your last link should be "http://technicaldiscovery.blogspot.com/" http://technicaldiscovery.blogspot.com/". There doesn't seem to be an active site at blogpost.com.
    - travisoliphant 13y ago
      Yes, thank you!. You have the correct link.
- donaldstufft 13y ago
  There is a way to make a virtualenv relocatable but it has some issues where it doesn't always work very well. I'm not sure offhand what those issues are though.
  - count 13y ago
    Isn't that basically docker?
    - mh- 13y ago
      no. oversimplifying here, but.. virtualenvs are just overrides to environment variables that change where things should look to find the python interpreter, libraries, modules, et al. docker is much more involved, using LXC to force process isolation, an effective chroot, networking restrictions, etc. many (most?) folks' current usage of virtualenv would not be portable to docker without substantive work to accomodate the restrictions that docker imposes. (note: this isn't a critique of docker; those things are features. they're just orthogonal to this)

deleted 13y ago

[deleted]

raptium 13y ago

I thought it was another web framework ...

r0muald 13y ago
Just like Ruby on Rails :)

mariocesar 13y ago

I use wheels to speed up the tests for sorl-thumbnail https://github.com/mariocesar/sorl-thumbnail/blob/master/.travis.yml#L26 https://github.com/mariocesar/sorl-thumbnail/blob/master/.tr... Initially I was into fully use wheel to install the environment but I found that some packages didn't work, now I just use it for just some critical packages and reduce the 12 minutes test to 3 minutes Will love to see wheels getting more attention, really feels like a great direction for python packaging.

jpdlla 13y ago
Love sorl-thumbnail, thanks for all the great work!
travisoliphant 13y ago
A lot of people in the PyData ecosystem using NumPy and Pandas are using conda and conda packcages (http://conda.pydata.org http://conda.pydata.org). Conda packages are easy to build and many exist at repo.continuum.io and on channels at http://binstar.org http://binstar.org

girvo 13y ago

For a language that prides itself on simplicity, this seems really painful...

pak 13y ago
Yes, it is. Coming to this mess from the joy of bundler/gems, npm, or hell, even CPAN will ruin your afternoon. It is just an unfortunate circumstance of the community not efficiently converging on well designed standards quickly enough, and so, the rocky soil has sprouted strange twisty crops. There are articles on how the setuptools/distutils frankenhack pretty much set off the whole trainwreck that you see today, e.g.: http://blog.startifact.com/posts/older/a-history-of-python-packaging.html http://blog.startifact.com/posts/older/a-history-of-python-p... Armin (the OP) has also written about this sad history previously: http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/ http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
- mixmastamyk 13y ago
  Would be nice if there were one packaging tool all those languages were using instead of each one reinventing it.
  - rmc 13y ago
    Something that integrated into your operating system distribution perhaps....
    - mh- 13y ago
      aptly put. you might be onto something.
      - mwexler 13y ago
        Groan. Well put, but still, groan. 8-)
      - mitchty 13y ago

pippy 13y ago

I've had the exact same experience as the author of this post. Eggs are great if you just want to install a python library. Horrible if you want to modify the library or distribute a program. I personally use a mixture of setuptools and distutils2, and it's a confusing mix.

shadowmint 13y ago

Wow, great article. I've heard of wheels for months, but I've never really understood why I should remotely care about them before (it always seemed to me they solved a problem that didn't exist; namely, binary distributions for pure python packages, since you can't support c-plugin packages cleanly with wheel anyhow...). Turns out the answer is: 1) Yes, you can kind of support packages with c-plugins, if you're specific and careful. 2) They make server deployments significantly faster. That's actually pretty neat. ...but damn, they still look like a massive pain to work with.

brabram 13y ago
I confirm about the pain. They should be transparent and you shouldn't need to care about them but that's absolutely not the case. Also, it's the responsibility of the pypi maintener to push wheels on it, but none do this, so you end up building your own wheels while this should have already been done on pypi and it's stupid. And one last thing: https://twitter.com/mitsuhiko/status/426700148409135104 https://twitter.com/mitsuhiko/status/426700148409135104 "It turns out, Python binary wheels on Python 2 are rejected by PyPI, I suppose because of the lack of UCS tag in the filename." Apart from all of this pain, I'm very happy to be able to use them. I'm really waiting for someone to build a "wheel on demand" website that would be a proxy to pypi. Or to turn pypi into a wheel farm (I don't know how realist this idea is).
- donaldstufft 13y ago
  <- PyPI Administrator. I have plans for either turning PyPI into a build farm, or making a secondary service that acts as a build farm for PyPI. We've just been more focused on cleaning up other issues.
themartorana 13y ago
After being bummed by the removal of 'bundle' in pip and writing an article about how we moved to using wheel for faster deployments[1], we moved to using Packer for server builds for our cloud infrastructure. Now we can backup to using a simple 'requirements.txt' file and not worrying about install and compile speeds. Removing 'bundle' was painful for us, and I even contributed a bunch of code to 'pip2pi' [2] to make it easy to set up an S3 bucket as a "local" pip mirror for our EC2 infrastructure, but it was all bandaid after bandaid. So far, Packer has felt much more elegant (despite the effort it took to put it in to our workflow) than wheel, pip2pi, or any other solution. [1] http://tech.flyclops.com/replacing-pip-bundle-374 http://tech.flyclops.com/replacing-pip-bundle-374 [2] https://github.com/wolever/pip2pi https://github.com/wolever/pip2pi (Edited for grammar.)

e12e 13y ago

A great introduction to wheels. I'm still not sure if wheels really is a good idea though. For simple use-cases traditonal pip seems fine, for more complex use-cases buildout seems better? Having tried to install numpy and pygame in a virtualenv I do see a need for something better -- but I'm not convinced wheels will solve the problem of complex c-library dependencies...?

_jb 13y ago

Google cache link: http://webcache.googleusercontent.com/search?q=cache:0rftZaFa274J:lucumr.pocoo.org/2014/1/27/python-on-wheels/+&cd=1&hl=en&ct=clnk&gl=ca http://webcache.googleusercontent.com/search?q=cache:0rftZaF...

beggi 13y ago

Page is currently down, so here: http://webcache.googleusercontent.com/search?q=cache:http://lucumr.pocoo.org/2014/1/27/python-on-wheels/ http://webcache.googleusercontent.com/search?q=cache:http://...

amouat 13y ago

On a slight tangent; does anyone else think that Docker is probably a better solution than virtualenv in most cases? I never really got on withe virtualenv.

dagw 13y ago
The subset of problems that virtualenv solves that are also solvable by Docker is pretty tiny. But sure if you have problem that can easily be solved by Docker, then Docker is probably a better solution.
- amouat 13y ago
  Really? I was thinking you just set up a new Docker env for each python project, then you don't need to worry about conflicting with libs from other projects. What am I missing?
  - dagw 13y ago
    Docker only runs on a small subset of the platforms python runs on is the most obvious one. Can docker easily access the host filesystem or do you have to copy all your data into each docker env? Can I access my GPU for doing CUDA/OpenGL/OpenCL things from withing docker? Does docker play nicely with GUI apps? Calling into a running docker env from an external program (when you are using an app that has python as scripting or plug-in language) doesn't work as far as I know. The overhead of setting up, tearing down an switching between several Docker envs seems to higher than with tools like virtualenvwrapper or conda. Now there may be workarounds for these (and all the other) problems but on the surface they don't seem easier than virtualenv.
    - amouat 13y ago
      Ok, thanks for the reply. Good questions. I was mainly thinking in the context of developing webapps. I'm interested in this enough that I intend to investigate and answer all your questions, perhaps in a blog post. I don't know enough right now to answer with certainty. However: - it will currently only work well if you are developing on Linux. MacOS isn't bad but requires vagrant/virtualbox. - I suspect switching between Docker envs may be faster, easier and cleaner than virtualenv which is why I suggested it. - There is definitely support for sharing data.

overgard 13y ago

Ugh. I'm so sick of the open source communities' disdain for binaries. There's no reason we should be recompiling shit everywhere. It's a waste of time, headspace, and processing power.

Pxtl 13y ago
When you're trying to support multiple architectures and OSes and your ABI took a decade or two to settle down, and all your tools are open-source so you have the source anyways, I can see the instinct to just say "I'll just compile it against the target machine" even if it's absurdly wasteful.
- hackerboos 13y ago
  Popular binaries can be uploaded falling back to building from source.

DrJ 13y ago

what about pex?

travisoliphant 13y ago

This is a nice post, and the Python packaging community is making strides in the right direction. However, it's not there yet. Armin could bring himself to say this in this post: "It's there, it kinda works, and it's probably worth your time." Conda is not perfect (it needs more people building conda binaries as well as a few pull requests to add https improvements) --- but I will confidently say "it is there, it does work now, and it is worth your time" This is especially true with a package repository like exists at http://repo.continuum.io http://repo.continuum.io and the many personal repositories at http://binstar.org http://binstar.org. People that have used conda for deployment (especially of complex C-dependencies in the NumPy and PyData stack) are saving time and effort today.

asmeurer 13y ago

"What the command is supposed to do is to collect all the dependencies and the convert them into wheels if necessary..." How does it convert the non-wheel packages into wheels?