5 ms·

How To Package Your Python Code

157 points by storborg 14y ago

JulianWasTaken 14y ago
Taking a quick look through this it looks pretty good! I find that beginners have tons of trouble navigating the slightly tricky waters that is packaging in Python, so more resources is great. One thing I think is missing is to point out not to use setuptools, but to use the less broken, more maintained, drop in replacement distribute (http://guide.python-distribute.org/ http://guide.python-distribute.org/). That website also has some further information on the packaging ecosystem that's worth flipping through.
- storborg 14y ago
  Thanks, I'll definitely add that to the "See Also" section. Can you elaborate on the places where setuptools is broken? I wanted to keep things as simple as possible for the sake of this tutorial, and I think the described constraints should avoid any setuptools bugs.
  - JulianWasTaken 14y ago
    There are various posts on the distribute/setuptools/distutils mailing list. I don't keep track personally, but I do know that it's pretty widely accepted at this point to use distribute.
  - deleted 14y ago
    [deleted]
- borntyping 14y ago
  Doesn't distribute miss out on a few of setuptools really useful features, like develop installs?
  - JulianWasTaken 14y ago
    It has develop. I rarely use it, but it's there. AFAIK it is not missing any features that setuptools has but I stick to the simple stuff.
gvalkov 14y ago
I've wondered why using a dictionary to initialize setuptools.setup() isn't more advocated in such guides. I understand this is superficial, but imho, this looks much clearer than the author's suggestion [1]: from setuptools import setup kw = { 'name' : 'funniest', 'version' : '0.1', 'description' : 'The funniest joke in the world', 'url' : 'http://github.com/storborg/funniest', 'author' : 'Flying Circus', 'author_email' : 'flyingcircus@example.com', 'license' : 'MIT', 'packages' : ['funniest'], 'zip_safe' : False, } if __name__ == '__main__': setup(**kw) Here are a few of my setup.py files that follow this convention [2] [3] [4]. [1]: http://www.scotttorborg.com/python-packaging/minimal.html#creating-the-scaffolding http://www.scotttorborg.com/python-packaging/minimal.html#cr... [2]: https://github.com/gvalkov/harstats-graphite/blob/master/setup.py https://github.com/gvalkov/harstats-graphite/blob/master/set... [3]: https://github.com/gvalkov/jenkins-autojobs/blob/master/setup.py https://github.com/gvalkov/jenkins-autojobs/blob/master/setu... [4]: https://github.com/gvalkov/python-evdev/blob/master/setup.py https://github.com/gvalkov/python-evdev/blob/master/setup.py
- borntyping 14y ago
  Because the only difference there is neater indentation, which you could do using the keywords method. Additionally, there's no point checking that the script is the main file, as there's only ever one use-case for the script, and that's to call setuptools.
  - gvalkov 14y ago
    It actually comes in handy if you're using Sphinx and don't want to maintain release and version in conf.py separately from the version in setup.py. There's no reasons why the data should be consumable only by setuptools .
    - borntyping 14y ago
      I put the version information inside the package, then anything with access to the package (which both setup.py and sphinx have) can access it - quite useful for doing things like checking the version number from the REPL, for instance.
maxjaderberg 14y ago
ive been writing python for ages but never found a simple introduction to how to package modules - always thought it was voodoo, so thanks a lot! so simple...
- indiecore 14y ago
  It is but now you know the correct incantations ~computers~ :)
borntyping 14y ago
While I've already been doing things a similar way, it's nice to have a clean, clear explanation of how to package stuff.
mixedbit 14y ago
Are setuptools suitable for packaging the whole web application (for example in Django) together with html/js/css files, configs for uwsgi and some management scripts? I once tried to do something like this and failed and I had an impression that setuptools are not really intended for such things, but mainly for packaging plain Python modules. Is this a right impression? If yes, what is a good way to package the whole Python based web application?
- borntyping 14y ago
  It sounds like you're trying to package an entire application, not a python package. If thats the case, maintaining and distributing it may be simplest to do with a git repository.
  - mercurial 14y ago
    In my limited experience, packaging works very well with Flask: http://flask.pocoo.org/docs/patterns/distribute/ http://flask.pocoo.org/docs/patterns/distribute/ You just need a quickly whipped up Fabric script, and then you deploy remotely from the command line.
- storborg 14y ago
  Although setuptools is capable of that kind of scenario, sometimes it's not worth the overhead. A good rule of thumb is that if you have defined access points for others to consume your code with (command line tools, Python modules, or Setuptools entry points), it's a good idea to make a Python package like this. Otherwise, skip it.
- grosskur 14y ago
  I find setuptools works great for packaging whole apps with HTML/JS/CSS. You can put these data files in your Python package directory, list them in MANIFEST.in, and set 'include_package_data=True' in setup.py. Then you can reference them in your code relative to __file__. And you can include a run script for your app using 'scripts' in setup.py, or let setuptools generate it using 'entry_points'. Take a look at https://github.com/getsentry/sentry https://github.com/getsentry/sentry for a good example. After doing 'pip install sentry', you can run the included gunicorn-based HTTP server via bin/sentry ('sentry init && sentry start'). A uWSGI-based server would have worked, too. I like to create a virtualenv under /opt, install the package and its dependencies with pip, and then run fpm to create an RPM or DEB. Then I use Chef or Puppet to install the package, deploy config files, deploy a service script (for upstart or daemontools), and enable the service. See also http://www.12factor.net/ http://www.12factor.net/ for some principles on how to structure your app. These can be applied whether or not you use setuptools. For example, Heroku's Python buildpack will deploy your 12-factor app in-place using requirements.txt and Procfile in your app root, ignoring setup.py. I find 12-factor apps are easier to deploy whatever method you choose, since configs/services/logs are cleanly separated from your app itself.
chewxy 14y ago
Here are other good packaging guides for Python (by Tarek Ziade). I think Scott Torbog has joined the ranks of these greats: http://www.aosabook.org/en/packaging.html http://www.aosabook.org/en/packaging.html http://guide.python-distribute.org/ http://guide.python-distribute.org/
rhettg 14y ago
You guys might also find this github project useful: https://github.com/splaice/py-bootstrap https://github.com/splaice/py-bootstrap
feniv 14y ago
I know this isn't exactly related to packaging the code, but any thoughts on utilities like py2exe (http://www.py2exe.org/ http://www.py2exe.org/), cx_Freeze (http://cx-freeze.sourceforge.net/ http://cx-freeze.sourceforge.net/) or pyinstaller (http://www.pyinstaller.org/ http://www.pyinstaller.org/) for distributing finished python modules to end-users?
dearmash 14y ago
I apologize for the spam, but apparently I'm unable to save articles for some reason. I'd like to read this later, and I want to see if a comment will work.
- doesnt_know 14y ago
  If only there was some way to save a url within the software that we use to browse the web. That would be an incredibly useful feature, someone here should consider developing an addon or put in a feature request for such a thing.
  - dearmash 14y ago
    Touche. Rather than ask other people if there exist issues with the intended story saving and voting features, I will start using browser bookmarks. Thank you for your valuable feedback. Pre-emtpive: "I know this is not the proper place to file a 'bug-report'" *edit: not alone (semi-old) http://news.ycombinator.com/item?id=4164468 http://news.ycombinator.com/item?id=4164468
phryk 14y ago
Thanks, this is exactly what I was looking for the last few weeks and couldn't find :)
pak 14y ago
The fact that there is no good, canonical guide for doing this in Python is one of the reasons I moved away from it and toward Ruby. With Ruby I had no problems writing and forking gems using the canonical guide (http://guides.rubygems.org/make-your-own-gem/ http://guides.rubygems.org/make-your-own-gem/). With Python, navigating the hell that is packaging is just demoralizing when it comes time to give end users a complicated chunk of code. (Setuptools is truly "a hack on a top of a bad design", not my own words.) I hope for Python's own sake that distribute (http://packages.python.org/distribute/ http://packages.python.org/distribute/) catches on and people are simply shamed out of using older tools, but since Python already has a problem with getting everyone to migrate code away from old things, I'm not holding my breath. I'm still not sure how to do certain things "right", e.g., distribute a modified version of somebody else's package with my own package. In Ruby, I can do this with bundler specifying a git repo for my own fork of the gem (and then submit a pull request for the original fork, if it's a patch that's useful upstream).
- cdavid 14y ago
  if setuptools is a hack on top of bad design, distribute has exactly the same issues, since it is simply a fork that was started because of the impression that setuptools dev was stalled. None of the (numerous) design issues from setuptools have been fixed in distribute.
norlowski 14y ago
Awesome job. I had been stressing about getting my code up there but your tutorial made it easy.

157 points by storborg 14y ago

Taking a quick look through this it looks pretty good! I find that beginners have tons of trouble navigating the slightly tricky waters that is packaging in Python, so more resources is great. One thing I think is missing is to point out not to use setuptools, but to use the less broken, more maintained, drop in replacement distribute (http://guide.python-distribute.org/ http://guide.python-distribute.org/). That website also has some further information on the packaging ecosystem that's worth flipping through.

storborg 14y ago
Thanks, I'll definitely add that to the "See Also" section. Can you elaborate on the places where setuptools is broken? I wanted to keep things as simple as possible for the sake of this tutorial, and I think the described constraints should avoid any setuptools bugs.
- JulianWasTaken 14y ago
  There are various posts on the distribute/setuptools/distutils mailing list. I don't keep track personally, but I do know that it's pretty widely accepted at this point to use distribute.
- deleted 14y ago
  [deleted]
borntyping 14y ago
Doesn't distribute miss out on a few of setuptools really useful features, like develop installs?
- JulianWasTaken 14y ago
  It has develop. I rarely use it, but it's there. AFAIK it is not missing any features that setuptools has but I stick to the simple stuff.

gvalkov 14y ago

I've wondered why using a dictionary to initialize setuptools.setup() isn't more advocated in such guides. I understand this is superficial, but imho, this looks much clearer than the author's suggestion [1]: from setuptools import setup kw = { 'name' : 'funniest', 'version' : '0.1', 'description' : 'The funniest joke in the world', 'url' : 'http://github.com/storborg/funniest', 'author' : 'Flying Circus', 'author_email' : 'flyingcircus@example.com', 'license' : 'MIT', 'packages' : ['funniest'], 'zip_safe' : False, } if __name__ == '__main__': setup(**kw) Here are a few of my setup.py files that follow this convention [2] [3] [4]. [1]: http://www.scotttorborg.com/python-packaging/minimal.html#creating-the-scaffolding http://www.scotttorborg.com/python-packaging/minimal.html#cr... [2]: https://github.com/gvalkov/harstats-graphite/blob/master/setup.py https://github.com/gvalkov/harstats-graphite/blob/master/set... [3]: https://github.com/gvalkov/jenkins-autojobs/blob/master/setup.py https://github.com/gvalkov/jenkins-autojobs/blob/master/setu... [4]: https://github.com/gvalkov/python-evdev/blob/master/setup.py https://github.com/gvalkov/python-evdev/blob/master/setup.py

borntyping 14y ago
Because the only difference there is neater indentation, which you could do using the keywords method. Additionally, there's no point checking that the script is the main file, as there's only ever one use-case for the script, and that's to call setuptools.
- gvalkov 14y ago
  It actually comes in handy if you're using Sphinx and don't want to maintain release and version in conf.py separately from the version in setup.py. There's no reasons why the data should be consumable only by setuptools .
  - borntyping 14y ago
    I put the version information inside the package, then anything with access to the package (which both setup.py and sphinx have) can access it - quite useful for doing things like checking the version number from the REPL, for instance.

maxjaderberg 14y ago

ive been writing python for ages but never found a simple introduction to how to package modules - always thought it was voodoo, so thanks a lot! so simple...

indiecore 14y ago
It is but now you know the correct incantations ~computers~ :)

borntyping 14y ago

While I've already been doing things a similar way, it's nice to have a clean, clear explanation of how to package stuff.

mixedbit 14y ago

Are setuptools suitable for packaging the whole web application (for example in Django) together with html/js/css files, configs for uwsgi and some management scripts? I once tried to do something like this and failed and I had an impression that setuptools are not really intended for such things, but mainly for packaging plain Python modules. Is this a right impression? If yes, what is a good way to package the whole Python based web application?

borntyping 14y ago
It sounds like you're trying to package an entire application, not a python package. If thats the case, maintaining and distributing it may be simplest to do with a git repository.
- mercurial 14y ago
  In my limited experience, packaging works very well with Flask: http://flask.pocoo.org/docs/patterns/distribute/ http://flask.pocoo.org/docs/patterns/distribute/ You just need a quickly whipped up Fabric script, and then you deploy remotely from the command line.
storborg 14y ago
Although setuptools is capable of that kind of scenario, sometimes it's not worth the overhead. A good rule of thumb is that if you have defined access points for others to consume your code with (command line tools, Python modules, or Setuptools entry points), it's a good idea to make a Python package like this. Otherwise, skip it.
grosskur 14y ago
I find setuptools works great for packaging whole apps with HTML/JS/CSS. You can put these data files in your Python package directory, list them in MANIFEST.in, and set 'include_package_data=True' in setup.py. Then you can reference them in your code relative to __file__. And you can include a run script for your app using 'scripts' in setup.py, or let setuptools generate it using 'entry_points'. Take a look at https://github.com/getsentry/sentry https://github.com/getsentry/sentry for a good example. After doing 'pip install sentry', you can run the included gunicorn-based HTTP server via bin/sentry ('sentry init && sentry start'). A uWSGI-based server would have worked, too. I like to create a virtualenv under /opt, install the package and its dependencies with pip, and then run fpm to create an RPM or DEB. Then I use Chef or Puppet to install the package, deploy config files, deploy a service script (for upstart or daemontools), and enable the service. See also http://www.12factor.net/ http://www.12factor.net/ for some principles on how to structure your app. These can be applied whether or not you use setuptools. For example, Heroku's Python buildpack will deploy your 12-factor app in-place using requirements.txt and Procfile in your app root, ignoring setup.py. I find 12-factor apps are easier to deploy whatever method you choose, since configs/services/logs are cleanly separated from your app itself.

chewxy 14y ago

Here are other good packaging guides for Python (by Tarek Ziade). I think Scott Torbog has joined the ranks of these greats: http://www.aosabook.org/en/packaging.html http://www.aosabook.org/en/packaging.html http://guide.python-distribute.org/ http://guide.python-distribute.org/

rhettg 14y ago

You guys might also find this github project useful: https://github.com/splaice/py-bootstrap https://github.com/splaice/py-bootstrap

feniv 14y ago

I know this isn't exactly related to packaging the code, but any thoughts on utilities like py2exe (http://www.py2exe.org/ http://www.py2exe.org/), cx_Freeze (http://cx-freeze.sourceforge.net/ http://cx-freeze.sourceforge.net/) or pyinstaller (http://www.pyinstaller.org/ http://www.pyinstaller.org/) for distributing finished python modules to end-users?

dearmash 14y ago

I apologize for the spam, but apparently I'm unable to save articles for some reason. I'd like to read this later, and I want to see if a comment will work.

doesnt_know 14y ago
If only there was some way to save a url within the software that we use to browse the web. That would be an incredibly useful feature, someone here should consider developing an addon or put in a feature request for such a thing.
- dearmash 14y ago
  Touche. Rather than ask other people if there exist issues with the intended story saving and voting features, I will start using browser bookmarks. Thank you for your valuable feedback. Pre-emtpive: "I know this is not the proper place to file a 'bug-report'" *edit: not alone (semi-old) http://news.ycombinator.com/item?id=4164468 http://news.ycombinator.com/item?id=4164468

phryk 14y ago

Thanks, this is exactly what I was looking for the last few weeks and couldn't find :)

pak 14y ago

The fact that there is no good, canonical guide for doing this in Python is one of the reasons I moved away from it and toward Ruby. With Ruby I had no problems writing and forking gems using the canonical guide (http://guides.rubygems.org/make-your-own-gem/ http://guides.rubygems.org/make-your-own-gem/). With Python, navigating the hell that is packaging is just demoralizing when it comes time to give end users a complicated chunk of code. (Setuptools is truly "a hack on a top of a bad design", not my own words.) I hope for Python's own sake that distribute (http://packages.python.org/distribute/ http://packages.python.org/distribute/) catches on and people are simply shamed out of using older tools, but since Python already has a problem with getting everyone to migrate code away from old things, I'm not holding my breath. I'm still not sure how to do certain things "right", e.g., distribute a modified version of somebody else's package with my own package. In Ruby, I can do this with bundler specifying a git repo for my own fork of the gem (and then submit a pull request for the original fork, if it's a patch that's useful upstream).

cdavid 14y ago
if setuptools is a hack on top of bad design, distribute has exactly the same issues, since it is simply a fork that was started because of the impression that setuptools dev was stalled. None of the (numerous) design issues from setuptools have been fixed in distribute.

norlowski 14y ago

Awesome job. I had been stressing about getting my code up there but your tutorial made it easy.