6 ms·

Django Production Deployment and Development Using Git

73 points by jknupp 15y ago

stargazer83 15y ago
We manage the deployments of our Django application, in multiple environments (production/staging/experimental/development) using Buildout. It probably is quite difficult to learn at first, but very rewarding once you get the hang of it. Deployment is all about being confident. And the confidence that Buildout has given us, is uncomparable to anything else. A cool way to produce various settings/configurations, is using the Buildout recipe collective.recipe.genshi. This enables us to create settings based on templates, using buildout configuration parameters. Very flexible solution!
spleeyah 15y ago
As a Django developer/sysadmin myself, I'd be very interested to hear more details about how you deploy.
- jknupp 15y ago
  Well, like it says in the post, my root directory served by Apache is just a clone of my git master branch. When I've finished making changes and committed them to master, deployment is a simple "git pull" and Apache reload via postinstall hook. Sometimes migrations need to be run, but they're all staged via another postinstall hook.
  - joelhaasnoot 15y ago
    Hmm, have you actually got everything working that way? I've found there's a few too many tasks to do that way, and wrote a Fabric script to deploy. It pulls git, checks for dependencies with a requirements file, compiles translations, runs collectstatic to put static files in the right places, and reloads Apache (and could run tests). Works quite well.
    - andrus 15y ago
      How does Fabric compare to setting up a git hook on post-merge to run collectstatic, etc.?
      - joelhaasnoot 15y ago
        Not too big a fan of git hooks, but mostly because I haven't played around with them too much. My script is here: https://gist.github.com/1755101 https://gist.github.com/1755101 A bunch of simple functions, some wrappers for directories and the environment, and an order of operations
  - spleeyah 15y ago
    Have you looked into things like Fabric for automating installations like this? It sounds like you're only deploying to one (or a few) machines, or is it more? This part of the development cycle has always interested me, especially to see how other people do things.
    - jsvaughan 15y ago
      I use Fabric for my deploys. The process is something like: * Check out to clean directory * Run tests * Zip source * Upload * Unzip * Backup existing (in case of needing rollback) * Deploy static files * Run South db migrations * Restart It's very easy to get started with Fabric and once you're using it, everything can easily be automated.
      - spleeyah 15y ago
        Cool! I use Puppet at work currently to do deployment stuff, but I've always been curious about other technologies like Fabric, et al. I'll probably look into them for personal projects or just for fun, eventually. It's on my [long and growing] list of things-to-do.
      - marcofucci 15y ago
        I use fabric as well with your same approach but I sometimes find easier to have git installed on the server and just pull from the main repository. I also use fabric for nginx and Apache config files so that I keep track of the changes and replace the files on the server if needed. And finally, I use fabric again to install needed virtualenv requirements.
    - jknupp 15y ago
      I've looked at Fabric but decided that, for now, it's a bit heavyweight for my needs since I'm only deploying to one machine. If my deployments get more complicated, I'll definitely take another look.
jpk 15y ago
I've been doing roughly this for a little while now, and it works pretty well. Particularly the bit about splitting the settings file. There are a few ways to do that, here[1]. [1] https://code.djangoproject.com/wiki/SplitSettings https://code.djangoproject.com/wiki/SplitSettings
hsparikh 15y ago
I am learning django/web development, and will likely be launching my app in the wild soon. A link that details out how to manage deployment and source code management using git would be tremendously helpful.
- jknupp 15y ago
  Is there anything from the post you'd like more detail on? Email me at <myusername>@gmail.com and I'd be happy to go over everything I've done so far regarding deployment and source code management via git.
  - hsparikh 15y ago
    Thank you! I will be in touch.
- kmfrk 15y ago
  This article by the Heroku guys also does a good job of explaining the core concepts of a modular approach to writing your app: http://www.12factor.net http://www.12factor.net.
rlander 15y ago
The global settings file is the single biggest hurdle, not only in django deployment, but also development. And it's been like this since I can first remember with no visible progress. It sucks to keep track of dev/prod local files, it makes it hard to setup a simple script inside of a django environment, it makes multi tenant apps unnecessarily complex, app developers usually want to shove every single configuration option inside of it... If anyone with more than a few hours of django development is doing split settings or something more complex, and even the official docs and best practices recommend doing so, why are we still stuck with it? There are much saner ways to deal with configuration: Flask's[1] is one nice way, as is Padrino's[2]. As a Django dev, this is one of those things that actually makes me jealous of Rails, where the core devs are not afraid of breaking backwards compatibility for sanity's sake. In my opinion it should be priority #1 for an overhaul. [1] http://flask.pocoo.org/docs/config/ http://flask.pocoo.org/docs/config/ [2] http://www.padrinorb.com/guides/mounting-applications http://www.padrinorb.com/guides/mounting-applications
- roam 15y ago
  I... don't understand your problem. Is it a PITA to split settings files every single time you start a new project? Yes, sure. Does it take more than five minutes to do so? No. Really, Flask's config options are better, but they aren't that different from Django. Use the default settings, tell Django which settings to use from the commandline or specify an environment variable pointing at the settings to use. Where you put those settings is completely up to you - nobody's saying they absolutely positively must be part of your repo. When you're working with multiple environments you'll have to specify which configuration to use one way or another so I'd prefer it to use the least amount of abstraction.
  - rlander 15y ago
    If you, me and anyone who's spent more than a few hours with Django split settings.py into multiple files (and heavily modify it), why is it still shipped as one monolithic file? Isn't the framework supposed to make our lives easier? Splitting settings into multiple files is the least of my problems (nowadays it is not a problem at all). The biggest problem I have with it is that it is global, which means I cannot safely modify it at runtime.
Osmose 15y ago
At Mozilla, we use a homegrown template called Playdoh[1][2] that includes split settings file, some sample deployment scripts, and lots of useful libraries that handle stuff like secure cookies, localization, strong password hashing, etc. I'm not super-familiar with our deployment process, but I do know that we have internal git repos that hold our production and dev environments (like local settings) and use puppet for setting up servers. [1]: Github: https://github.com/mozilla/playdoh https://github.com/mozilla/playdoh [2]: Docs: http://playdoh.readthedocs.org/en/latest/index.html http://playdoh.readthedocs.org/en/latest/index.html
- jpk 15y ago
  Puppet is pretty cool. I recently used it to do provisioning in a Vagrant[1] VM to duplicate my development environment for other members of my team. I'll probably use it for production stuff soon, too. [1] http://vagrantup.com/ http://vagrantup.com/

73 points by jknupp 15y ago

We manage the deployments of our Django application, in multiple environments (production/staging/experimental/development) using Buildout. It probably is quite difficult to learn at first, but very rewarding once you get the hang of it. Deployment is all about being confident. And the confidence that Buildout has given us, is uncomparable to anything else. A cool way to produce various settings/configurations, is using the Buildout recipe collective.recipe.genshi. This enables us to create settings based on templates, using buildout configuration parameters. Very flexible solution!

spleeyah 15y ago

As a Django developer/sysadmin myself, I'd be very interested to hear more details about how you deploy.

jknupp 15y ago
Well, like it says in the post, my root directory served by Apache is just a clone of my git master branch. When I've finished making changes and committed them to master, deployment is a simple "git pull" and Apache reload via postinstall hook. Sometimes migrations need to be run, but they're all staged via another postinstall hook.
- joelhaasnoot 15y ago
  Hmm, have you actually got everything working that way? I've found there's a few too many tasks to do that way, and wrote a Fabric script to deploy. It pulls git, checks for dependencies with a requirements file, compiles translations, runs collectstatic to put static files in the right places, and reloads Apache (and could run tests). Works quite well.
  - andrus 15y ago
    How does Fabric compare to setting up a git hook on post-merge to run collectstatic, etc.?
    - joelhaasnoot 15y ago
      Not too big a fan of git hooks, but mostly because I haven't played around with them too much. My script is here: https://gist.github.com/1755101 https://gist.github.com/1755101 A bunch of simple functions, some wrappers for directories and the environment, and an order of operations
- spleeyah 15y ago
  Have you looked into things like Fabric for automating installations like this? It sounds like you're only deploying to one (or a few) machines, or is it more? This part of the development cycle has always interested me, especially to see how other people do things.
  - jsvaughan 15y ago
    I use Fabric for my deploys. The process is something like: * Check out to clean directory * Run tests * Zip source * Upload * Unzip * Backup existing (in case of needing rollback) * Deploy static files * Run South db migrations * Restart It's very easy to get started with Fabric and once you're using it, everything can easily be automated.
    - spleeyah 15y ago
      Cool! I use Puppet at work currently to do deployment stuff, but I've always been curious about other technologies like Fabric, et al. I'll probably look into them for personal projects or just for fun, eventually. It's on my [long and growing] list of things-to-do.
    - marcofucci 15y ago
      I use fabric as well with your same approach but I sometimes find easier to have git installed on the server and just pull from the main repository. I also use fabric for nginx and Apache config files so that I keep track of the changes and replace the files on the server if needed. And finally, I use fabric again to install needed virtualenv requirements.
  - jknupp 15y ago
    I've looked at Fabric but decided that, for now, it's a bit heavyweight for my needs since I'm only deploying to one machine. If my deployments get more complicated, I'll definitely take another look.

jpk 15y ago

I've been doing roughly this for a little while now, and it works pretty well. Particularly the bit about splitting the settings file. There are a few ways to do that, here[1]. [1] https://code.djangoproject.com/wiki/SplitSettings https://code.djangoproject.com/wiki/SplitSettings

hsparikh 15y ago

I am learning django/web development, and will likely be launching my app in the wild soon. A link that details out how to manage deployment and source code management using git would be tremendously helpful.

jknupp 15y ago
Is there anything from the post you'd like more detail on? Email me at <myusername>@gmail.com and I'd be happy to go over everything I've done so far regarding deployment and source code management via git.
- hsparikh 15y ago
  Thank you! I will be in touch.
kmfrk 15y ago
This article by the Heroku guys also does a good job of explaining the core concepts of a modular approach to writing your app: http://www.12factor.net http://www.12factor.net.

rlander 15y ago

The global settings file is the single biggest hurdle, not only in django deployment, but also development. And it's been like this since I can first remember with no visible progress. It sucks to keep track of dev/prod local files, it makes it hard to setup a simple script inside of a django environment, it makes multi tenant apps unnecessarily complex, app developers usually want to shove every single configuration option inside of it... If anyone with more than a few hours of django development is doing split settings or something more complex, and even the official docs and best practices recommend doing so, why are we still stuck with it? There are much saner ways to deal with configuration: Flask's[1] is one nice way, as is Padrino's[2]. As a Django dev, this is one of those things that actually makes me jealous of Rails, where the core devs are not afraid of breaking backwards compatibility for sanity's sake. In my opinion it should be priority #1 for an overhaul. [1] http://flask.pocoo.org/docs/config/ http://flask.pocoo.org/docs/config/ [2] http://www.padrinorb.com/guides/mounting-applications http://www.padrinorb.com/guides/mounting-applications

roam 15y ago
I... don't understand your problem. Is it a PITA to split settings files every single time you start a new project? Yes, sure. Does it take more than five minutes to do so? No. Really, Flask's config options are better, but they aren't that different from Django. Use the default settings, tell Django which settings to use from the commandline or specify an environment variable pointing at the settings to use. Where you put those settings is completely up to you - nobody's saying they absolutely positively must be part of your repo. When you're working with multiple environments you'll have to specify which configuration to use one way or another so I'd prefer it to use the least amount of abstraction.
- rlander 15y ago
  If you, me and anyone who's spent more than a few hours with Django split settings.py into multiple files (and heavily modify it), why is it still shipped as one monolithic file? Isn't the framework supposed to make our lives easier? Splitting settings into multiple files is the least of my problems (nowadays it is not a problem at all). The biggest problem I have with it is that it is global, which means I cannot safely modify it at runtime.

Osmose 15y ago

At Mozilla, we use a homegrown template called Playdoh[1][2] that includes split settings file, some sample deployment scripts, and lots of useful libraries that handle stuff like secure cookies, localization, strong password hashing, etc. I'm not super-familiar with our deployment process, but I do know that we have internal git repos that hold our production and dev environments (like local settings) and use puppet for setting up servers. [1]: Github: https://github.com/mozilla/playdoh https://github.com/mozilla/playdoh [2]: Docs: http://playdoh.readthedocs.org/en/latest/index.html http://playdoh.readthedocs.org/en/latest/index.html

jpk 15y ago
Puppet is pretty cool. I recently used it to do provisioning in a Vagrant[1] VM to duplicate my development environment for other members of my team. I'll probably use it for production stuff soon, too. [1] http://vagrantup.com/ http://vagrantup.com/