5 ms·

I'll use the opportunity to moan about the state of packaging in the Python world. Why, after so many years, there is no way for me to ship software written in

by majke 11y ago

I'll use the opportunity to moan about the state of packaging in the Python world. Why, after so many years, there is no way for me to ship software written in python, in deb format? I end up doing some real crazy stuff: * build in docker * using virtualenv /usr/local/mypackage * then manually copy over the init scripts "cp /usr/local/mypackage/bin/* /usr/local/bin" * and debianize "/usr/local/mypackage/" and "/usr/local/bin". Except it doesn't work: * I need root do create the venv in /usr/local. * This technique copies garbage like "activate" script to bin. * virtualenv ships its own python binary. If the python version on host differ, it won't work. * The workaround is to rewrite the bin scripts. Total mess. All I need is a deb with some bin scripts, and all dependencies bundled. Is it _that_ hard...

mtrn 11y ago
I tried a few things myself, like fpm[1], rpmvenv[2] and lately pex[3] - but so far, I haven't found something, that I would call solid and simple. We have constrained deployment environments that do not allow most of the tools, that would ease the process on a development machine. Basically, I need a deb or rpm. I have some hopes to wrap a pex into deb/rpm, but I would not call this approach simple. That's unfortunate since Python is a wonderful language for many data-sciency tasks - Python makes things possible and pleasant, that would be a pain in other languages. * [1] https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm * [2] https://github.com/kevinconway/rpmvenv https://github.com/kevinconway/rpmvenv * [3] https://pex.readthedocs.org/en/latest/ https://pex.readthedocs.org/en/latest/ * Another tool: https://github.com/spotify/dh-virtualenv https://github.com/spotify/dh-virtualenv
- jlees 11y ago
  We use dh-virtualenv and it's working out pretty well for us so far.
- vetrom 11y ago
  crohr's pkgr (pkgr.io) works well as well.
troymc 11y ago
Did you try conda?
AUmrysh 11y ago
You can force virtualenv to include a specific python version virtualenv -p python2.7 env_name or virtualenv --python=/some/path/to/python env_name You've made the best argument for using pip like this, however. Instead of having to install a virtualenvironment and install dependencies and all of that, you could just ship your program with the pip -t libraries, though you'd have to include the path somehow on the install so python knew where to find those files.
- majke 11y ago
  Well, your user might have python in /some/path/to/python. My user my have in /usr/bin/python or maybe /usr/local/bin/python.
- deleted 11y ago
  [deleted]
JoshTriplett 11y ago
> Why, after so many years, there is no way for me to ship software written in python, in deb format? How about using dh_python? And if you're building in docker anyway, why use virtualenv? > All I need is a deb with some bin scripts, and all dependencies bundled. Ah, that would be a problem. Python .debs are supposed to depend on separate python module packages, not bundle everything. For what you're doing, have you considered turning your entire program into a single executable Python zipfile? See youtube-dl for an example.
jaimebuelta 11y ago
I know that distutils supports to create RPM packages https://docs.python.org/2/distutils/builtdist.html#creating-rpm-packages https://docs.python.org/2/distutils/builtdist.html#creating-... I can't see why that cannot be adapted to .deb packages as well...
rcfox 11y ago
I don't know off hand if this handles dependencies, but I've used this in the past to create deb files: python setup.py --command-packages=stdeb.command bdist_deb I'm pretty sure this works without requiring some special module.
- majke 11y ago
  What about dependencies? Setuptools assume all is in place. That's why you need pip/virtualenv. So with that you expect your host to have all the stuff. Virtualenv installs some .so files in its thing, so venv directory is tightly coupled with the python binary it runs. Ie: you need to sort it out on the moment you build venv, and shipping venv directory is not correct.

by majke 11y ago

mtrn 11y ago

I tried a few things myself, like fpm[1], rpmvenv[2] and lately pex[3] - but so far, I haven't found something, that I would call solid and simple. We have constrained deployment environments that do not allow most of the tools, that would ease the process on a development machine. Basically, I need a deb or rpm. I have some hopes to wrap a pex into deb/rpm, but I would not call this approach simple. That's unfortunate since Python is a wonderful language for many data-sciency tasks - Python makes things possible and pleasant, that would be a pain in other languages. * [1] https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm * [2] https://github.com/kevinconway/rpmvenv https://github.com/kevinconway/rpmvenv * [3] https://pex.readthedocs.org/en/latest/ https://pex.readthedocs.org/en/latest/ * Another tool: https://github.com/spotify/dh-virtualenv https://github.com/spotify/dh-virtualenv

jlees 11y ago
We use dh-virtualenv and it's working out pretty well for us so far.
vetrom 11y ago
crohr's pkgr (pkgr.io) works well as well.

troymc 11y ago

Did you try conda?

AUmrysh 11y ago

You can force virtualenv to include a specific python version virtualenv -p python2.7 env_name or virtualenv --python=/some/path/to/python env_name You've made the best argument for using pip like this, however. Instead of having to install a virtualenvironment and install dependencies and all of that, you could just ship your program with the pip -t libraries, though you'd have to include the path somehow on the install so python knew where to find those files.

majke 11y ago
Well, your user might have python in /some/path/to/python. My user my have in /usr/bin/python or maybe /usr/local/bin/python.
deleted 11y ago
[deleted]

JoshTriplett 11y ago

> Why, after so many years, there is no way for me to ship software written in python, in deb format? How about using dh_python? And if you're building in docker anyway, why use virtualenv? > All I need is a deb with some bin scripts, and all dependencies bundled. Ah, that would be a problem. Python .debs are supposed to depend on separate python module packages, not bundle everything. For what you're doing, have you considered turning your entire program into a single executable Python zipfile? See youtube-dl for an example.

jaimebuelta 11y ago

I know that distutils supports to create RPM packages https://docs.python.org/2/distutils/builtdist.html#creating-rpm-packages https://docs.python.org/2/distutils/builtdist.html#creating-... I can't see why that cannot be adapted to .deb packages as well...

rcfox 11y ago

I don't know off hand if this handles dependencies, but I've used this in the past to create deb files: python setup.py --command-packages=stdeb.command bdist_deb I'm pretty sure this works without requiring some special module.

majke 11y ago
What about dependencies? Setuptools assume all is in place. That's why you need pip/virtualenv. So with that you expect your host to have all the stuff. Virtualenv installs some .so files in its thing, so venv directory is tightly coupled with the python binary it runs. Ie: you need to sort it out on the moment you build venv, and shipping venv directory is not correct.