9 ms·

The post is interesting, but it doesn't mention two major difficulties: zero downtime deploys and database migrations.

by ngrilly 11y ago

herge 11y ago
We use django migrations, and I still haven't found a way to do zero downtime deploys and migrations short of doing the following: If we are trying to deploy migration #1, first deploy a version B of the code that supports (but provides the same set of features as before) the db both after migration #1 and before migration #1 (maybe with the help of a feature flag that is set in the migration). Do the migration. The deploy a version C of the code which removed the feature check above. But all this requires 2 different versions of the code and a lot of process just to ship out one migration. It gets combinatorically worse if you have more than one migration to deploy.
- mwarkentin 11y ago
  Yep - unfortunately that seems to be the only way to handle it though.
- raziel2p 11y ago
  I wish there was a way to "pause" incoming requests in web servers. Most deployments (migrations + code) take less than a few seconds, and I'd be fine with some users having to wait 2 seconds for a request to finish over their request hitting a 500 (due to inconsistent code/database) or 503 (putting the site into maintenance mode).
  - acdha 11y ago
    One approach which can work pretty well in many apps is having something like a CDN or Varnish which can serve stale content when the backend is unreachable. That allows the code you need to bootstrap your app to be served as long as your edge cache is running and your JavaScript can some sort of retry+backoff for failed requests or even do things like check for image error states to trigger a reload.
  - tmedbury 11y ago
    Load Balancers.
  - sciurus 11y ago
    Have your load balancer drain connections to a web server. When it has no request in progress, deploy to it. When that's done, move on the the next web server.
    - singlow 11y ago
      This doesn't solve the problem when there is a DB migration.
      - sciurus 11y ago
        This is covered by herge's parent comment. Deploy code that can work with either the old or new DB structure before you perform the migration.
  - tpg 11y ago
    haproxy allows re-dispatching failed requests some number of times. If you have an extremely brief outage due to a deploy, redispatching failed requests 3 times may be sufficient. I imagine other load balancers have similar functionality.
  - wvenable 11y ago
    Usually we aren't deploying a schema change that's really huge so we just go for it and let the application crash for those users who happen to hit a place where the code/schema are out of sync. Zero (no crash) downtime deployments seems like too much effort for too little gain.
    - AznHisoka 11y ago
      Agreed. Seems like a case of optimization for technical completeness rather than a business need.
  - kl4m 11y ago
    Basecamp apparently uses an nginx/openresty plugin to pauses requests during a deployment: https://github.com/basecamp/intermission https://github.com/basecamp/intermission
  - netghost 11y ago
    Folks from Braintree gave a talk about how they did this. They'd queue up all the incoming requests in Redis, then replay them once the site was back up.
  - duggan 11y ago
    There is, sort of. If you're hosting on Linux, you can use iptables to artificially delay responses. That and some other methods are described in a post on the Yelp engineering blog[1]. 1: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...
    - sciurus 11y ago
      I don't think that's quite what raziel2p wants. Won't that still allow the web server to receive and process the request, but just delay it's response?
- jsmeaton 11y ago
  (I think I'm just expanding on what you say you already do - but I'd already written it out by the time I realised that this was exactly what you are doing, so I'm going to just leave it). If you can, always make migrations backwards compatible with the previous version of the code, so they don't need to be rolled back if the code needs to be rolled back. Having a good migration rollback procedure is nice too, but usually unnecessary if testing has gone well. If you need to add a new field to the model then always create it as nullable, whether or not you use a default value. That'll allow this particular database migration to run on previous versions of the code. You can test this by generating the migration SQL and running it on the database for your test/dev environment which is running your current production code. Once the new code and migration is running in production and you're satisfied with it, immediately create a new change which makes the previous migration more strict (remove nullable, use a data migration or default value for existing nulls). Test, stage, deploy that. Having backwards compatible migrations leads into zero downtime deploys also. Once the data migration has run, your app servers can be running version X or X-1. Before you push new code to a node, do a graceful shutdown (allowing queued requests to complete) or remove the node from the cluster (haproxy socket api for example), update the code, bring the node back into the cluster. Caveats: - not all database migrations can be backward compatible, but most can be made so by breaking your change into two or more changes. First - make the change without strict integrity checks. Second, enforce integrity checks and provide defaults. - zero downtime deploys requires multiple nodes (counter example would be welcome, but I can't think of one).
- bhuga 11y ago
  > and a lot of process just to ship out one migration. I would have agreed with you before working at GitHub, but the end result is that deploying is so easy that 3 deploys does not feel like "a lot of process." On most of our applications I can do this in 15 minutes or less.
holman 11y ago
Yeah; I tried to stay away from really low-level aspects (just because they're hard to generalize across languages), and also just because the damn thing was so long already, ha. :) As far as database migrations are concerned, GitHub (and others, of course) takes the perspective of migrating before the code that uses those migrations go out. In other words, as @herge says in a sibling comment, the code that gets pushed needs to support two branches of code and two branches of data simultaneously. It's certainly some extra work (and can be pretty gnarly depending on the scope of the migration), but once you get to a certain point it's kind of the only way to do no-downtime migrations. There's many possibilities to help with the actual migration process, depending on what database you're using. With MySQL, for example, you can do something like the process in lhm: https://github.com/soundcloud/lhm https://github.com/soundcloud/lhm Zero-downtime deploys aren't super difficult in Rubyland anymore (many have written how they achieve it in Unicorn, for example), although I'm not as familiar with how other folk do it across other languages and platforms these days.
- heavenlyhash 11y ago
  +1 -- treating the database and the backend as two separate services with their own release lifecycles and APIs is "obvious"... ...well, once someone says it out loud. :) After that, of course it makes sense that they would have a need for an API compatibility window across at least two versions. It's exactly the same issues as supporting a backend and client side where you can't instantaneously force an update to all clients. With your DB, you're in control of the version, but you're certainly not in control of making it "instantaneously", so the same rules apply as when you're waiting for some curmudgeon user to update: API versioning and a support window. Now, if only we had an automagic way to make it less painful for a project to support two different versions of an API from the same codebase....
mrighele 11y ago
For the part regarding database migrations, you can find here [1] an interesting podcast about SQL database (schema) evolution. Among the other things they talk about how to have multiple schema revision coexisting at the same time. While hopefully most of us just need to support at most two revisions at the same time, it gives a number of interesting ideas. [1] http://www.se-radio.net/2012/06/episode-186-martin-fowler-and-pramod-sadalage-on-agile-database-development/ http://www.se-radio.net/2012/06/episode-186-martin-fowler-an...

The post is interesting, but it doesn't mention two major difficulties: zero downtime deploys and database migrations.

by ngrilly 11y ago

The post is interesting, but it doesn't mention two major difficulties: zero downtime deploys and database migrations.

herge 11y ago

We use django migrations, and I still haven't found a way to do zero downtime deploys and migrations short of doing the following: If we are trying to deploy migration #1, first deploy a version B of the code that supports (but provides the same set of features as before) the db both after migration #1 and before migration #1 (maybe with the help of a feature flag that is set in the migration). Do the migration. The deploy a version C of the code which removed the feature check above. But all this requires 2 different versions of the code and a lot of process just to ship out one migration. It gets combinatorically worse if you have more than one migration to deploy.

mwarkentin 11y ago
Yep - unfortunately that seems to be the only way to handle it though.
raziel2p 11y ago
I wish there was a way to "pause" incoming requests in web servers. Most deployments (migrations + code) take less than a few seconds, and I'd be fine with some users having to wait 2 seconds for a request to finish over their request hitting a 500 (due to inconsistent code/database) or 503 (putting the site into maintenance mode).
- acdha 11y ago
  One approach which can work pretty well in many apps is having something like a CDN or Varnish which can serve stale content when the backend is unreachable. That allows the code you need to bootstrap your app to be served as long as your edge cache is running and your JavaScript can some sort of retry+backoff for failed requests or even do things like check for image error states to trigger a reload.
- tmedbury 11y ago
  Load Balancers.
- sciurus 11y ago
  Have your load balancer drain connections to a web server. When it has no request in progress, deploy to it. When that's done, move on the the next web server.
  - singlow 11y ago
    This doesn't solve the problem when there is a DB migration.
    - sciurus 11y ago
      This is covered by herge's parent comment. Deploy code that can work with either the old or new DB structure before you perform the migration.
- tpg 11y ago
  haproxy allows re-dispatching failed requests some number of times. If you have an extremely brief outage due to a deploy, redispatching failed requests 3 times may be sufficient. I imagine other load balancers have similar functionality.
- wvenable 11y ago
  Usually we aren't deploying a schema change that's really huge so we just go for it and let the application crash for those users who happen to hit a place where the code/schema are out of sync. Zero (no crash) downtime deployments seems like too much effort for too little gain.
  - AznHisoka 11y ago
    Agreed. Seems like a case of optimization for technical completeness rather than a business need.
- kl4m 11y ago
  Basecamp apparently uses an nginx/openresty plugin to pauses requests during a deployment: https://github.com/basecamp/intermission https://github.com/basecamp/intermission
- netghost 11y ago
  Folks from Braintree gave a talk about how they did this. They'd queue up all the incoming requests in Redis, then replay them once the site was back up.
- duggan 11y ago
  There is, sort of. If you're hosting on Linux, you can use iptables to artificially delay responses. That and some other methods are described in a post on the Yelp engineering blog[1]. 1: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...
  - sciurus 11y ago
    I don't think that's quite what raziel2p wants. Won't that still allow the web server to receive and process the request, but just delay it's response?
jsmeaton 11y ago
(I think I'm just expanding on what you say you already do - but I'd already written it out by the time I realised that this was exactly what you are doing, so I'm going to just leave it). If you can, always make migrations backwards compatible with the previous version of the code, so they don't need to be rolled back if the code needs to be rolled back. Having a good migration rollback procedure is nice too, but usually unnecessary if testing has gone well. If you need to add a new field to the model then always create it as nullable, whether or not you use a default value. That'll allow this particular database migration to run on previous versions of the code. You can test this by generating the migration SQL and running it on the database for your test/dev environment which is running your current production code. Once the new code and migration is running in production and you're satisfied with it, immediately create a new change which makes the previous migration more strict (remove nullable, use a data migration or default value for existing nulls). Test, stage, deploy that. Having backwards compatible migrations leads into zero downtime deploys also. Once the data migration has run, your app servers can be running version X or X-1. Before you push new code to a node, do a graceful shutdown (allowing queued requests to complete) or remove the node from the cluster (haproxy socket api for example), update the code, bring the node back into the cluster. Caveats: - not all database migrations can be backward compatible, but most can be made so by breaking your change into two or more changes. First - make the change without strict integrity checks. Second, enforce integrity checks and provide defaults. - zero downtime deploys requires multiple nodes (counter example would be welcome, but I can't think of one).
bhuga 11y ago
> and a lot of process just to ship out one migration. I would have agreed with you before working at GitHub, but the end result is that deploying is so easy that 3 deploys does not feel like "a lot of process." On most of our applications I can do this in 15 minutes or less.

holman 11y ago

Yeah; I tried to stay away from really low-level aspects (just because they're hard to generalize across languages), and also just because the damn thing was so long already, ha. :) As far as database migrations are concerned, GitHub (and others, of course) takes the perspective of migrating before the code that uses those migrations go out. In other words, as @herge says in a sibling comment, the code that gets pushed needs to support two branches of code and two branches of data simultaneously. It's certainly some extra work (and can be pretty gnarly depending on the scope of the migration), but once you get to a certain point it's kind of the only way to do no-downtime migrations. There's many possibilities to help with the actual migration process, depending on what database you're using. With MySQL, for example, you can do something like the process in lhm: https://github.com/soundcloud/lhm https://github.com/soundcloud/lhm Zero-downtime deploys aren't super difficult in Rubyland anymore (many have written how they achieve it in Unicorn, for example), although I'm not as familiar with how other folk do it across other languages and platforms these days.

heavenlyhash 11y ago
+1 -- treating the database and the backend as two separate services with their own release lifecycles and APIs is "obvious"... ...well, once someone says it out loud. :) After that, of course it makes sense that they would have a need for an API compatibility window across at least two versions. It's exactly the same issues as supporting a backend and client side where you can't instantaneously force an update to all clients. With your DB, you're in control of the version, but you're certainly not in control of making it "instantaneously", so the same rules apply as when you're waiting for some curmudgeon user to update: API versioning and a support window. Now, if only we had an automagic way to make it less painful for a project to support two different versions of an API from the same codebase....

mrighele 11y ago

For the part regarding database migrations, you can find here [1] an interesting podcast about SQL database (schema) evolution. Among the other things they talk about how to have multiple schema revision coexisting at the same time. While hopefully most of us just need to support at most two revisions at the same time, it gives a number of interesting ideas. [1] http://www.se-radio.net/2012/06/episode-186-martin-fowler-and-pramod-sadalage-on-agile-database-development/ http://www.se-radio.net/2012/06/episode-186-martin-fowler-an...