6 ms·

Can someone briefly explain how ZDD works in general? I guess both versions of the app must be running simultaneously, with new traffic being routed to the new

Can someone briefly explain how ZDD works in general? I guess both versions of the app must be running simultaneously, with new traffic being routed to the new version of the app. But what about DB migrations? Assuming the app uses a single database, and the new version of the app introduces changes to the DB schema, the new app version would modify the schema during startup via a migration script. However, the previous version of the app still expects the old schema. How is that handled?

andrejguran 2y ago
Migrations have to be backwards compatible so the DB schema can serve both versions of the app. It's an extra price to pay for having ZDD or rolling deployments and something to keep in mind. But it's generally done by all the larger companies
efortis 2y ago
Yes, both versions must be running at some point. The load balancer starts accepting connections on Server2 and stops accepting new connections on Server1. Then, Server1 disconnects when all of its connections are closed. It could be different Servers or multiple Workers on one server. During that window, as the other comments said, migrations have to be backwards compatible.
diggan 2y ago
First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with current and future schema - Verify everything still works - Run migrations - Verify everything still works - Clean up the acquired technical debt (the code that worked with the schema that no longer exists) at some point, or run out of runway and it won't be an issue
- wejick 2y ago
  This is very good explanation, no judgment and simply educational. Appreciated Though I'm still surprised that some people run DB alteration on application start up. Never saw one in real life.
  - diggan 2y ago
    > Though I'm still surprised that some people run DB alteration on application start up I think I've seen it more commonly in the Golang ecosystem, for some reason. Also not sure how common it is nowadays, but seen lots of deployments (contained in Ansible scripts, Makefiles, Bash scripts or whatever) where the migration+deploy is run directly in sequence automatically for each deploy, rather than as discrete steps. Edit: Maybe it's more of an educational problem than something else, where learning resources either don't specify when to actually run migrations or straight up recommend people to run migrations on application startup (one example: https://articles.wesionary.team/integrating-migration-tool-in-gin-framework-golang-139676bc4af7 https://articles.wesionary.team/integrating-migration-tool-i...)
  - miki123211 2y ago
    It makes things somewhat easier if your app is smallish and your workflow is something like e.g. Github Actions automatically deploying all commits on main to Fly or Render.
  - whartung 2y ago
    We do this. It has worked very well for us. There's a couple of fundamental rules to follow. First, don't put something that will have insane impact into the application deploy changes. 99% of the DB changes are very cheap, and very minor. If the deploy is going to be very expensive, then just don't do it, we'll do it out of band. This has not been a problem in practice with our 20ish person team. Second, it was kind of like double entry accounting. Once you committed the change, you can not go back and "fix it". If you did something really wrong (i.e. see above), then sure, but if not, you commit a correcting entry instead. Because you don't know who has recently downloaded your commit, and run it against their database. The changes are a list of incremental steps that the system applies in order, if they had not been applied before. So, they are treated as, essentially, append only. And it has worked really well for us, keeping the diverse developers who deploy again local databases in sync with little drama. I've incorporated the same concept in my GUI programs that stand up their own DB. It's a very simple system.
stephenr 2y ago
Others have described the how part if you do need truly zero downtime deployments, but I think it's worth pointing out that for most organisations, and most migrations, the amount of downtime due to a db migration is virtually indistinguishable from zero, particularly if you have a regional audience, and can aim for "quiet" hours to perform deployments.
- diggan 2y ago
  > the amount of downtime due to a db migration is virtually indistinguishable from zero Besides, once you've run a service for a while that has acquired enough data for migrations to take a while, you realize that there are in fact two different types of migrations. "Schema migrations" which are generally fast and "Data migrations" that depending on the amount of data can take seconds or days. Or you can do the "data migrations" when needed (on the fly) instead of processing all the data. Can get gnarly quickly though. Splitting those also allows you to reduce maintenance downtime if you don't have zero-downtime deployments already.
  - stephenr 2y ago
    Very much so, we handle these very differently for $client. Schema migrations are versioned in git with the app, with up/down (or forward/reverse) migration scripts and are applied automatically during deployment of the associated code change to a given environment. SQL Data migrations are stored in git so we have a record but are never applied automatically, always manually. The other thing we've used along these lines, is having one or more low priority job(s) added to a queue, to apply some kind of change to records. These are essentially still data migrations, but they're written as part of the application code base (as a Job) rather than in SQL.
  - sgarland 2y ago
    Schema migrations can be quite lengthy, mostly if you made a mistake earlier. Some things that come to mind are changing a column’s type, or extending VARCHAR length (with caveats; under certain circumstances it’s instant).
gsanderson 2y ago
I haven't tried it but it looks like Xata has come up with a neat solution to DB migrations (at least for postgres). There can be two versions of the app running. https://xata.io/blog/multi-version-schema-migrations https://xata.io/blog/multi-version-schema-migrations
efxhoy 2y ago
Strong migrations helps writing migrations that are safe for ZDD deploys. We use it in our rails app, catches quite a few potential footguns. https://github.com/ankane/strong_migrations https://github.com/ankane/strong_migrations

andrejguran 2y ago

Migrations have to be backwards compatible so the DB schema can serve both versions of the app. It's an extra price to pay for having ZDD or rolling deployments and something to keep in mind. But it's generally done by all the larger companies

efortis 2y ago

Yes, both versions must be running at some point. The load balancer starts accepting connections on Server2 and stops accepting new connections on Server1. Then, Server1 disconnects when all of its connections are closed. It could be different Servers or multiple Workers on one server. During that window, as the other comments said, migrations have to be backwards compatible.

diggan 2y ago

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with current and future schema - Verify everything still works - Run migrations - Verify everything still works - Clean up the acquired technical debt (the code that worked with the schema that no longer exists) at some point, or run out of runway and it won't be an issue

wejick 2y ago
This is very good explanation, no judgment and simply educational. Appreciated Though I'm still surprised that some people run DB alteration on application start up. Never saw one in real life.
- diggan 2y ago
  > Though I'm still surprised that some people run DB alteration on application start up I think I've seen it more commonly in the Golang ecosystem, for some reason. Also not sure how common it is nowadays, but seen lots of deployments (contained in Ansible scripts, Makefiles, Bash scripts or whatever) where the migration+deploy is run directly in sequence automatically for each deploy, rather than as discrete steps. Edit: Maybe it's more of an educational problem than something else, where learning resources either don't specify when to actually run migrations or straight up recommend people to run migrations on application startup (one example: https://articles.wesionary.team/integrating-migration-tool-in-gin-framework-golang-139676bc4af7 https://articles.wesionary.team/integrating-migration-tool-i...)
- miki123211 2y ago
  It makes things somewhat easier if your app is smallish and your workflow is something like e.g. Github Actions automatically deploying all commits on main to Fly or Render.
- whartung 2y ago
  We do this. It has worked very well for us. There's a couple of fundamental rules to follow. First, don't put something that will have insane impact into the application deploy changes. 99% of the DB changes are very cheap, and very minor. If the deploy is going to be very expensive, then just don't do it, we'll do it out of band. This has not been a problem in practice with our 20ish person team. Second, it was kind of like double entry accounting. Once you committed the change, you can not go back and "fix it". If you did something really wrong (i.e. see above), then sure, but if not, you commit a correcting entry instead. Because you don't know who has recently downloaded your commit, and run it against their database. The changes are a list of incremental steps that the system applies in order, if they had not been applied before. So, they are treated as, essentially, append only. And it has worked really well for us, keeping the diverse developers who deploy again local databases in sync with little drama. I've incorporated the same concept in my GUI programs that stand up their own DB. It's a very simple system.

stephenr 2y ago

Others have described the how part if you do need truly zero downtime deployments, but I think it's worth pointing out that for most organisations, and most migrations, the amount of downtime due to a db migration is virtually indistinguishable from zero, particularly if you have a regional audience, and can aim for "quiet" hours to perform deployments.

diggan 2y ago
> the amount of downtime due to a db migration is virtually indistinguishable from zero Besides, once you've run a service for a while that has acquired enough data for migrations to take a while, you realize that there are in fact two different types of migrations. "Schema migrations" which are generally fast and "Data migrations" that depending on the amount of data can take seconds or days. Or you can do the "data migrations" when needed (on the fly) instead of processing all the data. Can get gnarly quickly though. Splitting those also allows you to reduce maintenance downtime if you don't have zero-downtime deployments already.
- stephenr 2y ago
  Very much so, we handle these very differently for $client. Schema migrations are versioned in git with the app, with up/down (or forward/reverse) migration scripts and are applied automatically during deployment of the associated code change to a given environment. SQL Data migrations are stored in git so we have a record but are never applied automatically, always manually. The other thing we've used along these lines, is having one or more low priority job(s) added to a queue, to apply some kind of change to records. These are essentially still data migrations, but they're written as part of the application code base (as a Job) rather than in SQL.
- sgarland 2y ago
  Schema migrations can be quite lengthy, mostly if you made a mistake earlier. Some things that come to mind are changing a column’s type, or extending VARCHAR length (with caveats; under certain circumstances it’s instant).

gsanderson 2y ago

I haven't tried it but it looks like Xata has come up with a neat solution to DB migrations (at least for postgres). There can be two versions of the app running. https://xata.io/blog/multi-version-schema-migrations https://xata.io/blog/multi-version-schema-migrations

efxhoy 2y ago

Strong migrations helps writing migrations that are safe for ZDD deploys. We use it in our rails app, catches quite a few potential footguns. https://github.com/ankane/strong_migrations https://github.com/ankane/strong_migrations