13 ms·

Python Deployment Anti-Patterns

293 points by craigkerstiens 15y ago

Pewpewarrows 15y ago
Regarding "Don't use ancient system Python versions" and "Use virtual environments", you can knock out two birds with one stone by just using pythonbrew. It also saves you the hassle of rolling your own deb/rpm if a package doesn't happen to exist. Also, Chef/Puppet aren't "alternatives" to something like Fabric. Use the former for server provisioning, and use the latter for actually kicking off the deployment process. Trying to shoe-horn the finer deployment steps (git checkout, tarballing, symlinks, building the virtualenv, etc) into Chef was a nightmare every time I tried. Those tasks are better suited for Fabric's imperative design. Plus you can just run any Chef commands from Fabric itself, or use something like pychef for finer grained control. It's a win/win.
- hynek 15y ago
  > Those tasks are better suited for Fabric's imperative design. Yes they are, but IMHO not on the target servers. I use Fabric to build DEBs that get deployed by Puppet. I prefer to have no build tools on target servers, YMMV.
  - mrsteveman1 15y ago
    DEBs for your own project files, or debs containing built python modules that the server needs to run those project files? If the latter, how well does that mix in with virtualenv? or do you just avoid it entirely?
    - hynek 15y ago
      I’m the dude that wrote article, so like it says: Own stuff + deps. Your can re-initialize a virtualenv to fix it by simply running virtualenv again. But pinky swear I’ll write the second article. ;)
      - bebop 15y ago
        Get on it! :) I really enjoyed the article and want to know more of your magic.
  - themanr 15y ago
    Fabric isn't normally deployed on target servers though - it connects to them through SSH.
    - hynek 15y ago
      That’s not what I meant: If you build your virtualenv on the target server, you need build tools like GCC or development files like libpq-dev. I prefer to have as few stuff on servers as possible.
      - themanr 15y ago
        Thanks - that does sound like a big advantage of the deb / rpm route. Plus you don't really want your app servers spending their CPU time compiling.
        hynek 15y ago
        That’s a plus too. An “aptitude dist-upgrade” is really fast.
        St-Clock 15y ago
        Yes, but unless rpm/deb works with softlinks, it is not atomic right? Do you do anything before/after dist-upgrade or is it included in the pre/post install script? For example, for our deployment, we rely on softlinks and uwsgi robust reload behavior to avoid losing requests. I've seen many devops who were using hg update/git update as a way to "deploy" (arg!), but I'm not sure about the behavior of deb/rpm.
        hynek 15y ago
        You can do whatever you want inside of the post-install hooks which are just shell scripts. Including any kind of soft link black magic. :) And you’re right: replacing files of a running application can lead to all kind of weirdness. I’d even prefer to lose some requests than to risk that.
- ch0wn 15y ago
  > [...] you can knock out two birds with one stone by just using pythonbrew Would you recommend using pythonbrew on a production system?
- deleted 15y ago
  [deleted]
- stephen_mcd 15y ago
  Fabric seems to be the most popular deployment tool, yet the author advises against it without giving any reason why. I'd love to see some proper detail in the article around why. And not in the vein of "Chef/Puppet are better", but more along the line of "here's what can go wrong with Fabric".
  - hynek 15y ago
    Fabric is an invaluable deployment tool, but you have to know its boundaries for what it was intended. With Fabric you tell what to _do_ and with Puppet/Chef you define what the result should _look_ like. You define how a server should look like and it can make sure its true for 1000 servers. Or 10000. It’s not about Fabric vs. Puppet, but how to use both in the most efficient way.
    - grncdr 15y ago
      I find the line between when to use fabric and when to use puppet/chef is needing the permissions of a more privileged user. I prefer to deploy applications in the home directory of a dedicated user account with minimal privileges, and use fabric for installing updates and running application tasks. OTOH, I'd prefer to be using puppet for creating the user accounts, managing the installation and configuration of PostgreSQL, putting app configuration in a safe location, and so on. This comes down to my (maybe antiquated?) view of having multiple applications running on a single server.
      - hynek 15y ago
        That’s exactly how we do it too. The only difference is that we don’t install the app using Fabric. Instead, puppet installs the deb with the app into the directory. Maybe I should add a “running apps as root” anti-pattern, but this its 2012 after all, everyone should know that, right? :-/
    - antihero 15y ago
      It would be fantastic if someone were to write a tutorial how to deploy, for instance, a simple Flask+MySQL or Django+PostgreSQL application using these tools. I'm at a loss as to where to start.
djtriptych 15y ago
As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Manage production environments with virtualenv http://www.virtualenv.org/ - Manage Configuration with puppet and/or chef http://puppetlabs.com/ http://www.opscode.com/chef/ - Automate local and remote sys admin tasks with Fabric http://fabfile.org Other tips: - Don't restrict yourself to old Python versions to appease your tools / libs. - Strongly consider rolling your own DEB/RPMs for your Python application. Author also touted: - Celery for task management http://celeryproject.org/ - Twisted for event-based python. http://twistedmatrix.com/trac/ - nginx / gunicorn for your python web server stack http://www.nginx.com/ http://gunicorn.org/
- Ixiaus 15y ago
  Yes to all of this and I'll throw in my two cents on our web server setup which is nginx + uwsgi (which has served remarkably well).
  - lifeisstillgood 15y ago
    +1 to uWSGI and nginx. Now wsgi is built into nginx distribution, and uwsgi is gaining areputation for rock solid performance it's a nobrainer.
  - alexatkeplar 15y ago
    Likewise +1 to Nginx + uwsgi. In the various performance benchmarks* uwsgi beats gunicorn, and uwsgi comes bundled with Nginx now, so there's really no reason to bother with setting up gunicorn any more. * http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
    - 15y ago
ch0wn 15y ago
> The trick is to build a debian package (but it can be done using RPMs just as well) with the application and the whole virtualenv inside. I would love to read an article describing some best practices for doing that. I tried it once and found it extremely difficult, reverting to a git checkout + virtualenv kind of deployment.
- hynek 15y ago
  It will come, I promise.
- davvid 15y ago
  Check out git's "make rpm" target. https://github.com/gitster/git/blob/master/Makefile https://github.com/gitster/git/blob/master/Makefile Hosting your own apt/yum repo is pretty simple. Does anyone have an example of a similar "make deb" target they could share? I've heard of git-dpm and git-buildpackage but haven't used them extensively myself. They're the debian git packaging tools. http://wiki.debian.org/PackagingWithGit http://wiki.debian.org/PackagingWithGit
  - hynek 15y ago
    Have a look at fpm: https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm More details to come.
  - ch0wn 15y ago
    I'm familiar with building and hosting deps in general. The problem I had was that I tried to build debs for every single dependency which turned out to be really hard to automate. That's why I'm really interested in seeing an approach with the virtualenv embedded into the package.
postfuturist 15y ago
We do this, but also add a couple layers of safety between us and PyPI: 1. Run your own secure, local pypi clone with exact source versions of the packages you use. 2. The packages for production are built into RPMs from the local pypi. PyPI is great for discovery, getting things running quickly, and testing new versions, but you never want to rely on it, even for development.
- hynek 15y ago
  Yeah, plus it speeds package building up and protects against not-so-occasional pypi outages.
- jmlane 15y ago
  I'd really enjoy reading about how to setup a PyPI mirror like the one you use in your development/deployment workflow. It seems like a really good idea, considering I've had problems with PyPI at really inconvenient times in the past.
  - grncdr 15y ago
    I've used ClueReleaseManager to great effect in the past. It lazy loads from an upstream PyPI so you don't need to maintain a full mirror. If you use the same instance as your main PyPI mirror for developing and CI/deployment you never have to worry about syncing manually: any package you develop with will be in the cache by the time your code hits the CI server.
  - postfuturist 15y ago
    I think we use this: http://pypi.python.org/pypi/chishop/ http://pypi.python.org/pypi/chishop/ You just set it up on a local server, and upload packages the same way they are uploaded to real PyPI. python setup.py register sdist upload You can specify alternate PyPI server with a ~/.pypirc. There are probably other ways to do this. What's nice about this is you can upload your own private packages or your own personal forks of packages. We do both.
serverascode 15y ago
I don't think using virtualenv to jam everything into a big deb file is really a best practice. But at the end of the day, I do have to do a lot of that with application deployment, but I try to only go as far as packaging libraries (ie. gems, jars, python equiv) in the rpm/deb file. RHEL 6 is python 2.6.6, btw. What happens when there are vulns for your stack?
- hynek 15y ago
  > What happens when there are vulns for your stack? That’s a good point and the answer is: You have to monitor your dependencies of public services (that aren’t that many). But you have to do that anyway, because I can’t explain to our customers that their data has been hacked because Ubuntu/Red Hat didn’t update Django (fast enough).
  - serverascode 15y ago
    You make it sound like if you do one then you can do 100. Not the case.
    - hynek 15y ago
      My public services don’t have 100 dependencies and that’s on purpose. Relying on magic distribution fairies for all your libraries is a IMHO a false sense of security, YMMV. How do you make sure that whenever one of your dependencies gets updated that your daemons get restarted? And what do you do if you need a package that isn’t part of your distribution?
      - serverascode 15y ago
        Do you have your own linux distro?
smacktoward 15y ago
I don't get the negativity on using your distro's packages, at least from the staying-stable perspective. Any decent package manager should let you pin/hold critical packages on a particular version, so if "the next Ubuntu ships with a different SQLAlchemy by default" you just hold the SQLAlchemy package at the version you want and then ignore it until you're ready to make that move. 99% percent of the time when I hear people complaining about their distro's packages, the complaints are coming from the opposite direction -- they want to run something bleeding-edge and the distro doesn't have it yet. (This is the standard beef Rubyists have with Debian, for instance -- that code that just hit Github ten minutes ago isn't in Debian's repos yet.)
- heretohelp 15y ago
  They rarely have a modern enough package to be worthwhile. When you control the package pipeline, you have the ability to dictate how modern or conservative your software is. The problem is that most programming languages (especially Python and Ruby) are ecosystems unto themselves and often move at a much faster pace than any stable distro (or LTS) could keep up with. That's why we have gems and pip. Cf. Ubuntu LTS MongoDB default is like 1.2 or something. This is why I switched to using 10gen's repo.
  - smacktoward 15y ago
    Perhaps, but that's a completely different argument than the one TFA is making. Besides, in the Ubuntu case anyway, LTS is supposed to be old. The whole point of LTS releases is to let slow-moving institutions/enterprises sit on ancient packages for 3-5 years without having to worry about backporting security updates. If you want recent versions of packages LTS is precisely the wrong place for you to be.
    - heretohelp 15y ago
      LTS is the default distro for a lot of server and VPS providers. Often the latest VM images will be rather...unstable, even putting aside the instability of the distro itself.
jordanb 15y ago
This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the line? He seems to draw it further down the stack than most people (no system python, for instance) but he doesn't give his opinion on, for instance, using the system postgresql. Anyway, I personally would draw the line further up the stack than him, but take things on a case-by-case basis, and I don't really consider it an "anti-pattern." With regards to fabric vs. puppet, I understand the advantages of puppet when you have a complicated, hetrogenous deployment environment. But the majority of projects I've worked on have the operations model of a set of identically-configured application servers back-ended against a database server. For this configuration, what does puppet give you? If the author's argument is that the site may eventually outgrow that model, well, I can see puppet becoming necessary, but why not cross that bridge when you get to it?
- seliopou 15y ago
  Point of fact, he does give his opinion on the postgresql issue: "We also have to compile our own Apaches and MySQLs because we need that fine-grained control." But that's besides the point. I don't think he's arguing that software should be completely isolated form the deployment operating system. That would be absurd, since, as you pointed out, software has to interact with the OS at some level, i.e., to manage system resources. Just because the OS ships with a bunch of packages with specific versions doesn't mean you have to use them. And what I think the OP is saying is that to make your applications portable and easily deployable, you shouldn't.
  - jordanb 15y ago
    Good catch. Sorry I missed that. I suppose I've forgotten that a lot of people still use MySQL, even for python. :) The question "why even have an OS?" was a rhetorical one, meant to illustrate the fact that the argument here is over where one draws the line, between the dependencies you maintain yourself and those you outsource to your OS vendor. Unless you're deploying on Linux From Scratch then you are outsourcing to the vendor at some level, so the only question is, where is that level? That this guy maintains even his own web and database servers means wants/needs to control things at a much deeper level than is typical. And I think that's beautiful if it works for him. My point of disagreement is his use of the word "antipattern" to suggest that any other choice is wrong.
bickfordb 15y ago
Long-time Python user here. For a lot of my projects I write a shell script which builds all of the application dependencies (including services) into a project directory and run them all from there. It takes a little bit of work to get going --- especially when building a new service for the first time --- but I like that it side-steps language-specific packaging tools (particularly the half-baked Python ones) and lets me pin an applications dependencies and port to various environments (develop on Mac, deploy on Unix) almost exactly. Integrating with Puppet/Chef is just a matter of breaking up the shell script into pieces.
afhof 15y ago
Using tmux is a Python daemon antipattern? And then "there's so much wrong about this approach" that he doesn't bother explaining why? Isn't that why we are reading the article: because we want to know why? If the author is trying to convince people to change their habits, he is doing a crummy job. He comes across as elitist and "if you don't do it my way you're wrong".
- eclark 15y ago
  He's talking about creating /etc/init.d scripts that do something like: screen python manage.py gunicorn & That is an anti-pattern. Granted he didn't really explain it so well.
  - hynek 15y ago
    Honestly I didn’t think it would be necessary. :-/ Your example is btw the advanced version. Many people just ssh on the host, fire up screen and start their server inside.
- hynek 15y ago
  I’m sorry if that felt to you like that. I’ll consider rewording it. update I added some more context. I would never spit on my beloved tmux. :)
lifeisstillgood 15y ago
Regarding virtualenv, I have come to the conclusion that Linux containers are robust enough now (like freebsd jails say two or three years ago) that I don't need to virtualise just python - I can afford to have the whole server as a "virtualenv" - no need for that extra complexity just install into site packages. No conflicts because a whole instance is dedicated. Jails take this to the limit - one virtual machine, one process - say Bind. A vulnerability in Bind ? The attacker takes over ... Nothing.
- j_baker 15y ago
  Sorry, but I'm not sure I understand your logic. Using virtualenv adds extra complexity, but virtualizing the entire server doesn't? I mean, the only complexity using virtualenv adds is having to run the virtualenv process once. After that, you can still install to site-packages. You just have to install to a different site-packages directory. Besides that, it's worth pointing out that using a virtualenv is not a security precaution. It's a precaution to prevent mucking up the global python installation for other packages that run on it. Using linux containers to achieve this seems like overkill.
  - lifeisstillgood 15y ago
    A late reply but I don't just want the python environment virtualised - if it's important enough that I should section off python, then there is a good chance I will want to consider the whole box as a single unit, python, firewall rules, database what ever. I tend to think the unit of abstraction should not be the python process, but the server. This is a little easier to grasp when you think of BSD jails where essentially you can choose to only run those processes that actually mTter - it's less virtualised and os than pick and mix an os. Apologies for late reply - I guess I am straightening it out in my head more than telling anyone else.
minikomi 15y ago
Just a thank you for writing a positive, easy to follow overview with links to more in-depth information. I love when people boil down experience and serve it without a side dish of attitude.
cdavid 15y ago
I am glad this is working for the OP, but pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community. There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions): - lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages, and frankly, if you need to rely on packages which change API willingnily betwen e.g. 1.5 and 1.6, or if each of your service depends on a different version of some library, you have bigger problems anyway. - any sufficiently complex deployment will depend on things that are not python, at which point you need a solution that integrates multiple languages. That is, you re-creating what a distribution is all about. - virtualenv relies on sources, so if some of your dependences are in C, every deploy means compilation - I still have no idea how security is handled when you put everything in virtualenv See also http://bytes.com/topic/python/answers/841071-eggs-virtualenv-apt-best-practices http://bytes.com/topic/python/answers/841071-eggs-virtualenv...
- hynek 15y ago
  > pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community. Wow. :( > There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions): I’d love to hear them. > lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages, and frankly, if you need to rely on packages which change API willingnily betwen e.g. 1.5 and 1.6, or if each of your service depends on a different version of some library, you have bigger problems anyway. Well, you said there are possibilities of problems but that they shouldn’t matter in an ideal world. Maybe you’re okay to take the chances but I’m not. Every code I deploy has been tested rigorously against a certain set of versions and that is the only combination of dependencies I’m willing to consider “working”. UnitTests with different dependencies are just as worthless like integration/functional tests against sqllite instead of the same DB type as in production. There’s even the possibility that your code works because of a bug and when that one gets fixed, you app goes south because of some weird side-effect. > any sufficiently complex deployment will depend on things that are not python, at which point you need a solution that integrates multiple languages. That is, you re-creating what a distribution is all about. I’m not sure if I understand what you mean, but yes if you want to use certain features outside the Python ecosystem, you’ll have to buckle up and package them yourself too. “We can’t do that, package XYZ is missing/too old.” isn’t really a good excuse to not do something that is important/good for your business. And that’s one of the main points of the article. > virtualenv relies on sources, so if some of your dependences are in C, every deploy means compilation That’s wrong if you go the way described: The virtualenv is packaged with the code. Build tools don’t belong on production servers. > I still have no idea how security is handled when you put everything in virtualenv Just as everywhere else. If you think it’s okay to tell customers that their data has been hacked because debian was to slow to issue a fix, be my guest. We can’t afford that. What happens on my servers security wise is _my_ responsibility and using ancient versions of Python libraries just to be able to blame others for FUBARs is not a solution in my book.
njharman 15y ago
Virtualenv is a half solution and a hack. Use vagrant and VMs. There's a whole sea of libs and software that isn't "versioned" by pip/virtualenv. supervisord is the wrong solution. It answers the wrong question (is the process running). It's worse than useless in that it has given false positives. The right question is (is the process responding correctly). Use monit or something else that actually does what's needed.
- mh- 15y ago
  virtualenv isn't just used for testing, though. it's widely used to isolate applications' python environments/dependencies from one another.
- antihero 15y ago
  Why would you want to virtualise an entire system when your dependencies are restricted to a bunch of Python modules?
  - mrweasel 15y ago
    Personally I would just go for Virtualenv, but you'd be surprised if you know people do with virtualization. I've mostly seen it in Windows shops, but it's by no means limited to that. A common "anti-pattern" I've seen is to build a vmware image, or set of images, that run your "environment". Developers then spin up copies and request changes that they need be made to the "master copy". When it's time to test and deploy, the code is deployed to a vmware image, which is then sent to the testers. When everything checks out, the code is once again deploy to a new copy of the image, which is then promoted to production. On argument we've seen is that it makes it easy to do a rollback of the system, using vmware snapshots. It might be a sensible idea in some cases, I just thing it's a bit weird and some overhead. The worst use case of this I've seen is a Telco, where you needed to spin up as much 12 vms, depending one what you and your teams was working on. But then again that's the same company that read the Perforce guidelines on recommended setup and still decided to just pill all the projects in to one repository.
- hynek 15y ago
  Vagrant ist nice for testing but I’m not going to deploy services to VirtualBox (we use kvm for that). Of course there are dependencies outside of the Python ecosystem but their influence proved to be negligible till now. YMMV. supervisor is not a an alternative to monitoring, I never claimed that. But that’s a whole different story.
cageface 15y ago
The ability of Go to produce a single, self-contained executable is one of the biggest advantages it has over the "scripting" languages. It makes deployment so much simpler.
ma2rten 15y ago
Don’t run your daemons in a tmux/screen Wow, I always though of myself as an idiot for doing this. But that is for some not yet launched thing. Who on earth does this for a production website?
IgorPartola 15y ago
My two cents: I am a developer + ops person and deploy Python apps all the time. Typically they are Django and Tornado services. On top we also have a lot of daemons and a ton of library code. I agree with the OP on most points but do not on a few. First DO use packages that come with the OS. The OP says that you should not have the distro maintainers dictating what you use. I say, use what is widely available. It takes the headache out of a lot of your deployments. If you are looking for a library that converts foo to bar look in your distro's repos before going on GitHub. Your sysadmin will thank you. Second, DO NOT use virtualenv. It fixes the symptoms (Python's packaging system has many shortcomings such as inability to uninstall recursively, poor dependency management, lack of pre and post install scripts, etc.), but not the problem. Instead, use distro-appropriate packages. Integrate your app into the system. This way you will never end up running a daemon inside a screen session, etc. You also get the ability to very nicely manage dependencies and a clean separation between code and configuration. Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested. If apache feels like a ball of mud, take the time to understand how to cut it down to a minimum and configure it properly. When it comes to infrastructure, making boring choices leads to predictable performance and less headaches more often than not (at least in my experience).
- read_wharf 15y ago
  "First DO use packages that come with the OS." I'd go middle ground, and start here, but consider a self-built package where necessary. It depends in part on the focus of your distro. virtualenv. What problem does it solve? Different python version/environments? Wouldn't that be better solved with another (virtual) server? I understand if an extra $20/month is an issue, but otherwise ...
  - read_wharf 15y ago
    I have just been educated by reading more of this thread. I can see an obvious need for one virtualenv, so that you can separate your service and its needs from the system python and its needs. Beyond that my inclination would be to go more servers rather than more virtualenvs, but circumstances vary and my experience is narrow.

293 points by craigkerstiens 15y ago

Regarding "Don't use ancient system Python versions" and "Use virtual environments", you can knock out two birds with one stone by just using pythonbrew. It also saves you the hassle of rolling your own deb/rpm if a package doesn't happen to exist. Also, Chef/Puppet aren't "alternatives" to something like Fabric. Use the former for server provisioning, and use the latter for actually kicking off the deployment process. Trying to shoe-horn the finer deployment steps (git checkout, tarballing, symlinks, building the virtualenv, etc) into Chef was a nightmare every time I tried. Those tasks are better suited for Fabric's imperative design. Plus you can just run any Chef commands from Fabric itself, or use something like pychef for finer grained control. It's a win/win.

hynek 15y ago
> Those tasks are better suited for Fabric's imperative design. Yes they are, but IMHO not on the target servers. I use Fabric to build DEBs that get deployed by Puppet. I prefer to have no build tools on target servers, YMMV.
- mrsteveman1 15y ago
  DEBs for your own project files, or debs containing built python modules that the server needs to run those project files? If the latter, how well does that mix in with virtualenv? or do you just avoid it entirely?
  - hynek 15y ago
    I’m the dude that wrote article, so like it says: Own stuff + deps. Your can re-initialize a virtualenv to fix it by simply running virtualenv again. But pinky swear I’ll write the second article. ;)
    - bebop 15y ago
      Get on it! :) I really enjoyed the article and want to know more of your magic.
- themanr 15y ago
  Fabric isn't normally deployed on target servers though - it connects to them through SSH.
  - hynek 15y ago
    That’s not what I meant: If you build your virtualenv on the target server, you need build tools like GCC or development files like libpq-dev. I prefer to have as few stuff on servers as possible.
    - themanr 15y ago
      Thanks - that does sound like a big advantage of the deb / rpm route. Plus you don't really want your app servers spending their CPU time compiling.
      - hynek 15y ago
        That’s a plus too. An “aptitude dist-upgrade” is really fast.
        St-Clock 15y ago
        Yes, but unless rpm/deb works with softlinks, it is not atomic right? Do you do anything before/after dist-upgrade or is it included in the pre/post install script? For example, for our deployment, we rely on softlinks and uwsgi robust reload behavior to avoid losing requests. I've seen many devops who were using hg update/git update as a way to "deploy" (arg!), but I'm not sure about the behavior of deb/rpm.
        hynek 15y ago
        You can do whatever you want inside of the post-install hooks which are just shell scripts. Including any kind of soft link black magic. :) And you’re right: replacing files of a running application can lead to all kind of weirdness. I’d even prefer to lose some requests than to risk that.
ch0wn 15y ago
> [...] you can knock out two birds with one stone by just using pythonbrew Would you recommend using pythonbrew on a production system?
deleted 15y ago
[deleted]
stephen_mcd 15y ago
Fabric seems to be the most popular deployment tool, yet the author advises against it without giving any reason why. I'd love to see some proper detail in the article around why. And not in the vein of "Chef/Puppet are better", but more along the line of "here's what can go wrong with Fabric".
- hynek 15y ago
  Fabric is an invaluable deployment tool, but you have to know its boundaries for what it was intended. With Fabric you tell what to _do_ and with Puppet/Chef you define what the result should _look_ like. You define how a server should look like and it can make sure its true for 1000 servers. Or 10000. It’s not about Fabric vs. Puppet, but how to use both in the most efficient way.
  - grncdr 15y ago
    I find the line between when to use fabric and when to use puppet/chef is needing the permissions of a more privileged user. I prefer to deploy applications in the home directory of a dedicated user account with minimal privileges, and use fabric for installing updates and running application tasks. OTOH, I'd prefer to be using puppet for creating the user accounts, managing the installation and configuration of PostgreSQL, putting app configuration in a safe location, and so on. This comes down to my (maybe antiquated?) view of having multiple applications running on a single server.
    - hynek 15y ago
      That’s exactly how we do it too. The only difference is that we don’t install the app using Fabric. Instead, puppet installs the deb with the app into the directory. Maybe I should add a “running apps as root” anti-pattern, but this its 2012 after all, everyone should know that, right? :-/
  - antihero 15y ago
    It would be fantastic if someone were to write a tutorial how to deploy, for instance, a simple Flask+MySQL or Django+PostgreSQL application using these tools. I'm at a loss as to where to start.

djtriptych 15y ago

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Manage production environments with virtualenv http://www.virtualenv.org/ - Manage Configuration with puppet and/or chef http://puppetlabs.com/ http://www.opscode.com/chef/ - Automate local and remote sys admin tasks with Fabric http://fabfile.org Other tips: - Don't restrict yourself to old Python versions to appease your tools / libs. - Strongly consider rolling your own DEB/RPMs for your Python application. Author also touted: - Celery for task management http://celeryproject.org/ - Twisted for event-based python. http://twistedmatrix.com/trac/ - nginx / gunicorn for your python web server stack http://www.nginx.com/ http://gunicorn.org/

Ixiaus 15y ago
Yes to all of this and I'll throw in my two cents on our web server setup which is nginx + uwsgi (which has served remarkably well).
- lifeisstillgood 15y ago
  +1 to uWSGI and nginx. Now wsgi is built into nginx distribution, and uwsgi is gaining areputation for rock solid performance it's a nobrainer.
- alexatkeplar 15y ago
  Likewise +1 to Nginx + uwsgi. In the various performance benchmarks* uwsgi beats gunicorn, and uwsgi comes bundled with Nginx now, so there's really no reason to bother with setting up gunicorn any more. * http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi http://nichol.as/benchmark-of-python-web-servers http://nichol.as/benchmark-of-python-web-servers
  - 15y ago

ch0wn 15y ago

> The trick is to build a debian package (but it can be done using RPMs just as well) with the application and the whole virtualenv inside. I would love to read an article describing some best practices for doing that. I tried it once and found it extremely difficult, reverting to a git checkout + virtualenv kind of deployment.

hynek 15y ago
It will come, I promise.
davvid 15y ago
Check out git's "make rpm" target. https://github.com/gitster/git/blob/master/Makefile https://github.com/gitster/git/blob/master/Makefile Hosting your own apt/yum repo is pretty simple. Does anyone have an example of a similar "make deb" target they could share? I've heard of git-dpm and git-buildpackage but haven't used them extensively myself. They're the debian git packaging tools. http://wiki.debian.org/PackagingWithGit http://wiki.debian.org/PackagingWithGit
- hynek 15y ago
  Have a look at fpm: https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm More details to come.
- ch0wn 15y ago
  I'm familiar with building and hosting deps in general. The problem I had was that I tried to build debs for every single dependency which turned out to be really hard to automate. That's why I'm really interested in seeing an approach with the virtualenv embedded into the package.

postfuturist 15y ago

We do this, but also add a couple layers of safety between us and PyPI: 1. Run your own secure, local pypi clone with exact source versions of the packages you use. 2. The packages for production are built into RPMs from the local pypi. PyPI is great for discovery, getting things running quickly, and testing new versions, but you never want to rely on it, even for development.

hynek 15y ago
Yeah, plus it speeds package building up and protects against not-so-occasional pypi outages.
jmlane 15y ago
I'd really enjoy reading about how to setup a PyPI mirror like the one you use in your development/deployment workflow. It seems like a really good idea, considering I've had problems with PyPI at really inconvenient times in the past.
- grncdr 15y ago
  I've used ClueReleaseManager to great effect in the past. It lazy loads from an upstream PyPI so you don't need to maintain a full mirror. If you use the same instance as your main PyPI mirror for developing and CI/deployment you never have to worry about syncing manually: any package you develop with will be in the cache by the time your code hits the CI server.
- postfuturist 15y ago
  I think we use this: http://pypi.python.org/pypi/chishop/ http://pypi.python.org/pypi/chishop/ You just set it up on a local server, and upload packages the same way they are uploaded to real PyPI. python setup.py register sdist upload You can specify alternate PyPI server with a ~/.pypirc. There are probably other ways to do this. What's nice about this is you can upload your own private packages or your own personal forks of packages. We do both.

serverascode 15y ago

I don't think using virtualenv to jam everything into a big deb file is really a best practice. But at the end of the day, I do have to do a lot of that with application deployment, but I try to only go as far as packaging libraries (ie. gems, jars, python equiv) in the rpm/deb file. RHEL 6 is python 2.6.6, btw. What happens when there are vulns for your stack?

hynek 15y ago
> What happens when there are vulns for your stack? That’s a good point and the answer is: You have to monitor your dependencies of public services (that aren’t that many). But you have to do that anyway, because I can’t explain to our customers that their data has been hacked because Ubuntu/Red Hat didn’t update Django (fast enough).
- serverascode 15y ago
  You make it sound like if you do one then you can do 100. Not the case.
  - hynek 15y ago
    My public services don’t have 100 dependencies and that’s on purpose. Relying on magic distribution fairies for all your libraries is a IMHO a false sense of security, YMMV. How do you make sure that whenever one of your dependencies gets updated that your daemons get restarted? And what do you do if you need a package that isn’t part of your distribution?
    - serverascode 15y ago
      Do you have your own linux distro?

smacktoward 15y ago

I don't get the negativity on using your distro's packages, at least from the staying-stable perspective. Any decent package manager should let you pin/hold critical packages on a particular version, so if "the next Ubuntu ships with a different SQLAlchemy by default" you just hold the SQLAlchemy package at the version you want and then ignore it until you're ready to make that move. 99% percent of the time when I hear people complaining about their distro's packages, the complaints are coming from the opposite direction -- they want to run something bleeding-edge and the distro doesn't have it yet. (This is the standard beef Rubyists have with Debian, for instance -- that code that just hit Github ten minutes ago isn't in Debian's repos yet.)

heretohelp 15y ago
They rarely have a modern enough package to be worthwhile. When you control the package pipeline, you have the ability to dictate how modern or conservative your software is. The problem is that most programming languages (especially Python and Ruby) are ecosystems unto themselves and often move at a much faster pace than any stable distro (or LTS) could keep up with. That's why we have gems and pip. Cf. Ubuntu LTS MongoDB default is like 1.2 or something. This is why I switched to using 10gen's repo.
- smacktoward 15y ago
  Perhaps, but that's a completely different argument than the one TFA is making. Besides, in the Ubuntu case anyway, LTS is supposed to be old. The whole point of LTS releases is to let slow-moving institutions/enterprises sit on ancient packages for 3-5 years without having to worry about backporting security updates. If you want recent versions of packages LTS is precisely the wrong place for you to be.
  - heretohelp 15y ago
    LTS is the default distro for a lot of server and VPS providers. Often the latest VM images will be rather...unstable, even putting aside the instability of the distro itself.

jordanb 15y ago

This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the line? He seems to draw it further down the stack than most people (no system python, for instance) but he doesn't give his opinion on, for instance, using the system postgresql. Anyway, I personally would draw the line further up the stack than him, but take things on a case-by-case basis, and I don't really consider it an "anti-pattern." With regards to fabric vs. puppet, I understand the advantages of puppet when you have a complicated, hetrogenous deployment environment. But the majority of projects I've worked on have the operations model of a set of identically-configured application servers back-ended against a database server. For this configuration, what does puppet give you? If the author's argument is that the site may eventually outgrow that model, well, I can see puppet becoming necessary, but why not cross that bridge when you get to it?

seliopou 15y ago
Point of fact, he does give his opinion on the postgresql issue: "We also have to compile our own Apaches and MySQLs because we need that fine-grained control." But that's besides the point. I don't think he's arguing that software should be completely isolated form the deployment operating system. That would be absurd, since, as you pointed out, software has to interact with the OS at some level, i.e., to manage system resources. Just because the OS ships with a bunch of packages with specific versions doesn't mean you have to use them. And what I think the OP is saying is that to make your applications portable and easily deployable, you shouldn't.
- jordanb 15y ago
  Good catch. Sorry I missed that. I suppose I've forgotten that a lot of people still use MySQL, even for python. :) The question "why even have an OS?" was a rhetorical one, meant to illustrate the fact that the argument here is over where one draws the line, between the dependencies you maintain yourself and those you outsource to your OS vendor. Unless you're deploying on Linux From Scratch then you are outsourcing to the vendor at some level, so the only question is, where is that level? That this guy maintains even his own web and database servers means wants/needs to control things at a much deeper level than is typical. And I think that's beautiful if it works for him. My point of disagreement is his use of the word "antipattern" to suggest that any other choice is wrong.

bickfordb 15y ago

Long-time Python user here. For a lot of my projects I write a shell script which builds all of the application dependencies (including services) into a project directory and run them all from there. It takes a little bit of work to get going --- especially when building a new service for the first time --- but I like that it side-steps language-specific packaging tools (particularly the half-baked Python ones) and lets me pin an applications dependencies and port to various environments (develop on Mac, deploy on Unix) almost exactly. Integrating with Puppet/Chef is just a matter of breaking up the shell script into pieces.

afhof 15y ago

Using tmux is a Python daemon antipattern? And then "there's so much wrong about this approach" that he doesn't bother explaining why? Isn't that why we are reading the article: because we want to know why? If the author is trying to convince people to change their habits, he is doing a crummy job. He comes across as elitist and "if you don't do it my way you're wrong".

eclark 15y ago
He's talking about creating /etc/init.d scripts that do something like: screen python manage.py gunicorn & That is an anti-pattern. Granted he didn't really explain it so well.
- hynek 15y ago
  Honestly I didn’t think it would be necessary. :-/ Your example is btw the advanced version. Many people just ssh on the host, fire up screen and start their server inside.
hynek 15y ago
I’m sorry if that felt to you like that. I’ll consider rewording it. update I added some more context. I would never spit on my beloved tmux. :)

lifeisstillgood 15y ago

Regarding virtualenv, I have come to the conclusion that Linux containers are robust enough now (like freebsd jails say two or three years ago) that I don't need to virtualise just python - I can afford to have the whole server as a "virtualenv" - no need for that extra complexity just install into site packages. No conflicts because a whole instance is dedicated. Jails take this to the limit - one virtual machine, one process - say Bind. A vulnerability in Bind ? The attacker takes over ... Nothing.

j_baker 15y ago
Sorry, but I'm not sure I understand your logic. Using virtualenv adds extra complexity, but virtualizing the entire server doesn't? I mean, the only complexity using virtualenv adds is having to run the virtualenv process once. After that, you can still install to site-packages. You just have to install to a different site-packages directory. Besides that, it's worth pointing out that using a virtualenv is not a security precaution. It's a precaution to prevent mucking up the global python installation for other packages that run on it. Using linux containers to achieve this seems like overkill.
- lifeisstillgood 15y ago
  A late reply but I don't just want the python environment virtualised - if it's important enough that I should section off python, then there is a good chance I will want to consider the whole box as a single unit, python, firewall rules, database what ever. I tend to think the unit of abstraction should not be the python process, but the server. This is a little easier to grasp when you think of BSD jails where essentially you can choose to only run those processes that actually mTter - it's less virtualised and os than pick and mix an os. Apologies for late reply - I guess I am straightening it out in my head more than telling anyone else.

minikomi 15y ago

Just a thank you for writing a positive, easy to follow overview with links to more in-depth information. I love when people boil down experience and serve it without a side dish of attitude.

cdavid 15y ago

I am glad this is working for the OP, but pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community. There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions): - lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages, and frankly, if you need to rely on packages which change API willingnily betwen e.g. 1.5 and 1.6, or if each of your service depends on a different version of some library, you have bigger problems anyway. - any sufficiently complex deployment will depend on things that are not python, at which point you need a solution that integrates multiple languages. That is, you re-creating what a distribution is all about. - virtualenv relies on sources, so if some of your dependences are in C, every deploy means compilation - I still have no idea how security is handled when you put everything in virtualenv See also http://bytes.com/topic/python/answers/841071-eggs-virtualenv-apt-best-practices http://bytes.com/topic/python/answers/841071-eggs-virtualenv...

hynek 15y ago
> pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community. Wow. :( > There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions): I’d love to hear them. > lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages, and frankly, if you need to rely on packages which change API willingnily betwen e.g. 1.5 and 1.6, or if each of your service depends on a different version of some library, you have bigger problems anyway. Well, you said there are possibilities of problems but that they shouldn’t matter in an ideal world. Maybe you’re okay to take the chances but I’m not. Every code I deploy has been tested rigorously against a certain set of versions and that is the only combination of dependencies I’m willing to consider “working”. UnitTests with different dependencies are just as worthless like integration/functional tests against sqllite instead of the same DB type as in production. There’s even the possibility that your code works because of a bug and when that one gets fixed, you app goes south because of some weird side-effect. > any sufficiently complex deployment will depend on things that are not python, at which point you need a solution that integrates multiple languages. That is, you re-creating what a distribution is all about. I’m not sure if I understand what you mean, but yes if you want to use certain features outside the Python ecosystem, you’ll have to buckle up and package them yourself too. “We can’t do that, package XYZ is missing/too old.” isn’t really a good excuse to not do something that is important/good for your business. And that’s one of the main points of the article. > virtualenv relies on sources, so if some of your dependences are in C, every deploy means compilation That’s wrong if you go the way described: The virtualenv is packaged with the code. Build tools don’t belong on production servers. > I still have no idea how security is handled when you put everything in virtualenv Just as everywhere else. If you think it’s okay to tell customers that their data has been hacked because debian was to slow to issue a fix, be my guest. We can’t afford that. What happens on my servers security wise is _my_ responsibility and using ancient versions of Python libraries just to be able to blame others for FUBARs is not a solution in my book.

njharman 15y ago

Virtualenv is a half solution and a hack. Use vagrant and VMs. There's a whole sea of libs and software that isn't "versioned" by pip/virtualenv. supervisord is the wrong solution. It answers the wrong question (is the process running). It's worse than useless in that it has given false positives. The right question is (is the process responding correctly). Use monit or something else that actually does what's needed.

mh- 15y ago
virtualenv isn't just used for testing, though. it's widely used to isolate applications' python environments/dependencies from one another.
antihero 15y ago
Why would you want to virtualise an entire system when your dependencies are restricted to a bunch of Python modules?
- mrweasel 15y ago
  Personally I would just go for Virtualenv, but you'd be surprised if you know people do with virtualization. I've mostly seen it in Windows shops, but it's by no means limited to that. A common "anti-pattern" I've seen is to build a vmware image, or set of images, that run your "environment". Developers then spin up copies and request changes that they need be made to the "master copy". When it's time to test and deploy, the code is deployed to a vmware image, which is then sent to the testers. When everything checks out, the code is once again deploy to a new copy of the image, which is then promoted to production. On argument we've seen is that it makes it easy to do a rollback of the system, using vmware snapshots. It might be a sensible idea in some cases, I just thing it's a bit weird and some overhead. The worst use case of this I've seen is a Telco, where you needed to spin up as much 12 vms, depending one what you and your teams was working on. But then again that's the same company that read the Perforce guidelines on recommended setup and still decided to just pill all the projects in to one repository.
hynek 15y ago
Vagrant ist nice for testing but I’m not going to deploy services to VirtualBox (we use kvm for that). Of course there are dependencies outside of the Python ecosystem but their influence proved to be negligible till now. YMMV. supervisor is not a an alternative to monitoring, I never claimed that. But that’s a whole different story.

cageface 15y ago

The ability of Go to produce a single, self-contained executable is one of the biggest advantages it has over the "scripting" languages. It makes deployment so much simpler.

ma2rten 15y ago

Don’t run your daemons in a tmux/screen Wow, I always though of myself as an idiot for doing this. But that is for some not yet launched thing. Who on earth does this for a production website?

IgorPartola 15y ago

My two cents: I am a developer + ops person and deploy Python apps all the time. Typically they are Django and Tornado services. On top we also have a lot of daemons and a ton of library code. I agree with the OP on most points but do not on a few. First DO use packages that come with the OS. The OP says that you should not have the distro maintainers dictating what you use. I say, use what is widely available. It takes the headache out of a lot of your deployments. If you are looking for a library that converts foo to bar look in your distro's repos before going on GitHub. Your sysadmin will thank you. Second, DO NOT use virtualenv. It fixes the symptoms (Python's packaging system has many shortcomings such as inability to uninstall recursively, poor dependency management, lack of pre and post install scripts, etc.), but not the problem. Instead, use distro-appropriate packages. Integrate your app into the system. This way you will never end up running a daemon inside a screen session, etc. You also get the ability to very nicely manage dependencies and a clean separation between code and configuration. Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested. If apache feels like a ball of mud, take the time to understand how to cut it down to a minimum and configure it properly. When it comes to infrastructure, making boring choices leads to predictable performance and less headaches more often than not (at least in my experience).

read_wharf 15y ago
"First DO use packages that come with the OS." I'd go middle ground, and start here, but consider a self-built package where necessary. It depends in part on the focus of your distro. virtualenv. What problem does it solve? Different python version/environments? Wouldn't that be better solved with another (virtual) server? I understand if an extra $20/month is an issue, but otherwise ...
- read_wharf 15y ago
  I have just been educated by reading more of this thread. I can see an obvious need for one virtualenv, so that you can separate your service and its needs from the system python and its needs. Beyond that my inclination would be to go more servers rather than more virtualenvs, but circumstances vary and my experience is narrow.