5 ms·

Having no-down-time when doing migrations is not an easy topic. > These defer drops will happen at least 30 minutes after the particular migration referenced r

Having no-down-time when doing migrations is not an easy topic. > These defer drops will happen at least 30 minutes after the particular migration referenced ran (in the next migration cycle), giving us peace of mind that the new application code is in place. Pretty much that's what I've seen before. You do series of deployments + migration scripts. Usually : - Migrate with backward-compatible modifications of the schema - Release application logic that takes advantage of both migrated / to-be-deprecated tables - Possible extra migration to sync the new / old tables. - Release application logic that stops using the to-be-deprecated tables. - Migrate with destructive modifications to remove the old table. Usually that's a huge complicated process which might be replaced with one migration and a few minutes of downtime. Sadly some companies can't afford it, so they do such changes with weeks planing. I usually vote for having some planned non-working-hours downtime.

zombieprocesses 9y ago
> Having no-down-time when doing migrations is not an easy topic. Depending on the nature of the migration and the type of data and the type of service and how coupled everything is, it's impossible. Especially when you are dealing with financial data and transactions that you can't afford to screw up even a single one. > I usually vote for having some planned non-working-hours downtime. Yep. No other way around it. You tell your clients weeks/months ahead of the planned downtime. They'll understand and migrate in the middle of the night over a holiday weekend. But this is in regards to major structural migrations, not simple schema updates or changes.
- greenleafjacob 9y ago
  Are you saying no downtime is impossible or merely expensive? Erlang/OTP with relups is an example of a system that provides zero downtime online upgrade and rollback with zero downtime.
  - zombieprocesses 9y ago
    I'm talking about serious systematic migrations, not upgrades. In these instances, it is impossible to avoid downtime ( whether it be a few seconds, minutes or even hours ). It's one of the reasons why SLAs do not offer 100% uptimes.
    - theptip 9y ago
      Interested in a concrete example of where you see a migration being impossible without downtime; I'm not aware of Amazon or Google taking their APIs down for maintenance, which suggests that you can solve these problems with enough engineering spend. > depending on... how coupled everything is If you're just saying here that some systems aren't currently able to migrate in a hitless way, rather than claiming that it would never be possible for some systems, then I fully agree. Hitless migrations require a lot of work, and probably aren't worth the cost for many applications/systems.
      - stickfigure 9y ago
        Neither Amazon nor Google run off of an RDBMS. Relational databases are awesome, but it's much harder to achieve 100% uptime compared to schemaless, distributed, replicated data stores. You easily can take down your system with one innocuous-looking DDL migration that works great in test but grinds the machine to a halt on a production dataset.
        theptip 9y ago
        That's a good general point, but Youtube does use MySQL: https://www.mysql.com/customers/view/?id=750 https://www.mysql.com/customers/view/?id=750 http://highscalability.com/youtube-architecture http://highscalability.com/youtube-architecture
sebazzz 9y ago
Good alternative to downtime: read-only copy of application.
- pandler 9y ago
  As with most thing, it depends, right? For one, you’d have to actually implement a read-only version of your application, including creating an understandable user experience for that scenario. That’s an additional task per feature, though, as opposed to an additional task per migration as described by the article. Additionally, that could cause a lot of headaches for your users if you’re continuously deploying. Then again, I kind of question what you’re doing if you’re constantly running DB migrations for a user application.
taeric 9y ago
This is the same story as service dependencies, as well. If you've doubled up on this, then you can actually hide your database dependency behind a service contract. If done perfectly (high bar), then you take full ownership of the migration and nobody external to your services should have any knowledge/care of your database schema. Or your service implementation. Nor is this where this trick ends. It is almost literally how Intel survived the supposed superior instructionsets of RISK back in the CISC days. They turned their implementation contract of CISC into essentially a service ocntract of "the computer will execute this for you." Then, took advantage of the misdirection and were able to get many of the RISK benefits in their computers. Obviously, this can burn you at an extreme. That fun quote of another layer of abstraction, and all of that. I would be surprised if there is a prescriptive "this is how you do it" that is applicable to everyone. To note, if you only have your company as a client of either service or schema, it might be faster to just make the change in one shot. Let that sink in. The 5 step process outlined above requires manpower to execute. Don't ignore that.
- nugator 9y ago
  > That fun quote of another layer of abstraction... Haven't heard, please elaborate.
  - taeric 9y ago
    Apologies, I used the wrong word. I should have said indirection. The David Wheeler quote on this page: https://en.wikipedia.org/wiki/Indirection https://en.wikipedia.org/wiki/Indirection "All problems in computer science can be solved by another level of indirection"
carlosdp 9y ago
There's actually some precedent with very high volume services taking the minimal downtime approach to avoid the increased amount of development time. Take Airbnb for example [1]. They chose to take two weeks to migrate their Messaging database with a few minutes of downtime rather than take longer to do it with no downtime. In the article, they discuss how they thought about the tradeoffs. [1] https://medium.com/airbnb-engineering/how-we-partitioned-airbnb-s-main-database-in-two-weeks-55f7e006ff21 https://medium.com/airbnb-engineering/how-we-partitioned-air...
grogers 9y ago
Is it really much extra work to do the migration online instead of offline? It's just that second step where the application writes to both tables (and usually a step where reads switch to the new tables). Everything else is necessary for either method. In many situations taking downtime for it is more expensive for development. When taking downtime, often you need to be really damn sure it works beforehand so you don't take unnecessary down time. Having to roll back and try again later after fixing something (and eating more downtime) can be a bit embarrassing. When you are doing migrations without downtime you can take your time between each step and add additional validations that might be impractical when trying to minimize downtime. If you don't need to minimize downtime then maybe it doesn't really matter but that situation seems rare from my perspective.
- djrobstep 9y ago
  > In many situations taking downtime for it is more expensive for development. This is absolutely true, and the main reason for this is that the tooling sucks. Actually testing that your migration will result in your intended database state is difficult and thus people simply don't do it. I've attempted to remedy this situation by writing a diff tool for Postgres, so you can confirm your migrations will result in a matching schema, and autogenerate more correct migration scripts. I also talked more about this topic at last year's PostgresOpen conference: https://www.youtube.com/watch?v=xr498W8oMRo https://www.youtube.com/watch?v=xr498W8oMRo
deleted 9y ago
[deleted]
ransom1538 9y ago
My teams have never had any difficulty in schema changes. To be honest, we don't even think of it as an issue. 1) No deleting or alternating columns needed by any production code. Need something different? Create a new column. After weeks of this column not being needed, we use 2) to delete it - but it isn't a priority. 2) Adding/Modifying columns just use pt-schema-change. I have seen this tool transform billion row tables - it also supports slave master.

zombieprocesses 9y ago

> Having no-down-time when doing migrations is not an easy topic. Depending on the nature of the migration and the type of data and the type of service and how coupled everything is, it's impossible. Especially when you are dealing with financial data and transactions that you can't afford to screw up even a single one. > I usually vote for having some planned non-working-hours downtime. Yep. No other way around it. You tell your clients weeks/months ahead of the planned downtime. They'll understand and migrate in the middle of the night over a holiday weekend. But this is in regards to major structural migrations, not simple schema updates or changes.

greenleafjacob 9y ago
Are you saying no downtime is impossible or merely expensive? Erlang/OTP with relups is an example of a system that provides zero downtime online upgrade and rollback with zero downtime.
- zombieprocesses 9y ago
  I'm talking about serious systematic migrations, not upgrades. In these instances, it is impossible to avoid downtime ( whether it be a few seconds, minutes or even hours ). It's one of the reasons why SLAs do not offer 100% uptimes.
  - theptip 9y ago
    Interested in a concrete example of where you see a migration being impossible without downtime; I'm not aware of Amazon or Google taking their APIs down for maintenance, which suggests that you can solve these problems with enough engineering spend. > depending on... how coupled everything is If you're just saying here that some systems aren't currently able to migrate in a hitless way, rather than claiming that it would never be possible for some systems, then I fully agree. Hitless migrations require a lot of work, and probably aren't worth the cost for many applications/systems.
    - stickfigure 9y ago
      Neither Amazon nor Google run off of an RDBMS. Relational databases are awesome, but it's much harder to achieve 100% uptime compared to schemaless, distributed, replicated data stores. You easily can take down your system with one innocuous-looking DDL migration that works great in test but grinds the machine to a halt on a production dataset.
      - theptip 9y ago
        That's a good general point, but Youtube does use MySQL: https://www.mysql.com/customers/view/?id=750 https://www.mysql.com/customers/view/?id=750 http://highscalability.com/youtube-architecture http://highscalability.com/youtube-architecture

sebazzz 9y ago

Good alternative to downtime: read-only copy of application.

pandler 9y ago
As with most thing, it depends, right? For one, you’d have to actually implement a read-only version of your application, including creating an understandable user experience for that scenario. That’s an additional task per feature, though, as opposed to an additional task per migration as described by the article. Additionally, that could cause a lot of headaches for your users if you’re continuously deploying. Then again, I kind of question what you’re doing if you’re constantly running DB migrations for a user application.

taeric 9y ago

This is the same story as service dependencies, as well. If you've doubled up on this, then you can actually hide your database dependency behind a service contract. If done perfectly (high bar), then you take full ownership of the migration and nobody external to your services should have any knowledge/care of your database schema. Or your service implementation. Nor is this where this trick ends. It is almost literally how Intel survived the supposed superior instructionsets of RISK back in the CISC days. They turned their implementation contract of CISC into essentially a service ocntract of "the computer will execute this for you." Then, took advantage of the misdirection and were able to get many of the RISK benefits in their computers. Obviously, this can burn you at an extreme. That fun quote of another layer of abstraction, and all of that. I would be surprised if there is a prescriptive "this is how you do it" that is applicable to everyone. To note, if you only have your company as a client of either service or schema, it might be faster to just make the change in one shot. Let that sink in. The 5 step process outlined above requires manpower to execute. Don't ignore that.

nugator 9y ago
> That fun quote of another layer of abstraction... Haven't heard, please elaborate.
- taeric 9y ago
  Apologies, I used the wrong word. I should have said indirection. The David Wheeler quote on this page: https://en.wikipedia.org/wiki/Indirection https://en.wikipedia.org/wiki/Indirection "All problems in computer science can be solved by another level of indirection"

carlosdp 9y ago

There's actually some precedent with very high volume services taking the minimal downtime approach to avoid the increased amount of development time. Take Airbnb for example [1]. They chose to take two weeks to migrate their Messaging database with a few minutes of downtime rather than take longer to do it with no downtime. In the article, they discuss how they thought about the tradeoffs. [1] https://medium.com/airbnb-engineering/how-we-partitioned-airbnb-s-main-database-in-two-weeks-55f7e006ff21 https://medium.com/airbnb-engineering/how-we-partitioned-air...

grogers 9y ago

Is it really much extra work to do the migration online instead of offline? It's just that second step where the application writes to both tables (and usually a step where reads switch to the new tables). Everything else is necessary for either method. In many situations taking downtime for it is more expensive for development. When taking downtime, often you need to be really damn sure it works beforehand so you don't take unnecessary down time. Having to roll back and try again later after fixing something (and eating more downtime) can be a bit embarrassing. When you are doing migrations without downtime you can take your time between each step and add additional validations that might be impractical when trying to minimize downtime. If you don't need to minimize downtime then maybe it doesn't really matter but that situation seems rare from my perspective.

djrobstep 9y ago
> In many situations taking downtime for it is more expensive for development. This is absolutely true, and the main reason for this is that the tooling sucks. Actually testing that your migration will result in your intended database state is difficult and thus people simply don't do it. I've attempted to remedy this situation by writing a diff tool for Postgres, so you can confirm your migrations will result in a matching schema, and autogenerate more correct migration scripts. I also talked more about this topic at last year's PostgresOpen conference: https://www.youtube.com/watch?v=xr498W8oMRo https://www.youtube.com/watch?v=xr498W8oMRo

deleted 9y ago

[deleted]

ransom1538 9y ago

My teams have never had any difficulty in schema changes. To be honest, we don't even think of it as an issue. 1) No deleting or alternating columns needed by any production code. Need something different? Create a new column. After weeks of this column not being needed, we use 2) to delete it - but it isn't a priority. 2) Adding/Modifying columns just use pt-schema-change. I have seen this tool transform billion row tables - it also supports slave master.