3 ms·

This one-by-one migration nicely contains problems though if something goes wrong - only one customer affected at a time (or whatever your batch size was) I’ve

This one-by-one migration nicely contains problems though if something goes wrong - only one customer affected at a time (or whatever your batch size was) I’ve done deployments this way pretty much everywhere.

SOLAR_FIELDS 2y ago
Good call out, it definitely reduces the blast radius to do it this way. Poor man’s isolation
- deleted 2y ago
  [deleted]
julik 2y ago
Good point. Also: nothing makes it impossible to apply migrations to multiple tenants in parallel, this is pretty much only IO. I didn't have to since my tenant numbers were very low, but very doable.
tyre 2y ago
Do you then run app instances per user? Or how do your upstream control for some DBs being on different schemas? At least with Single Point of Success database design you either move or don’t.
- jasonthorsness 2y ago
  App per tenant would be nice but in practice mostly I’ve seen shared app instances with a careful dance of “schema changes can’t break app” and “app can’t use new schema until it is 100% rolled out” which is super annoying.
  - what 2y ago
    How is that super annoying? That’s exactly how migrations are supposed to be done, even without “app per tenant”.
hobs 2y ago
I think its ok if you have a traditional server approach, but in the cloud this is a really great way to make yourself pay a lot for a little. Most systems have a minimum amount of traffic they'll let you pay for, most object stores are not setup to stream small updates in a transactionally consistent way, there's a ton of complexity "in the small" as it were.
- stackskipton 2y ago
  Depends on the cloud and their model. Azure/Google have MySQL/PostGres options where you get a server they manage OS/Database software for but you can run as many databases as hardware will allow. Some of other cloud databases don't charge you for database but simply for usage so in that case, usage = customer revenue so cost should scale as you do.
netghost 2y ago
I'm curious, how do you handle being in a partial state? It seems like it could be operationally tricky to have some users migrated, some pending, and some in an error state. On the other hand, I could imagine doing lazy migrations as user or tenants sign in as a nice way to smooth out the increased effort in migrating if you can engineer for it.
- njs12345 2y ago
  When not per tenant you still have this problem, and typically resolve by releasing database migrations separately to the code that uses changes (ensuring backwards compatibility). I guess per-tenant you would handle similarly but build some tooling to monitor the state of migrations. It might even be easier in some ways, as you might be more able to take table locks when migrating per-tenant.
- magicalhippo 2y ago
  We've got a desktop B2B application which we also run with per-tenant DB. This makes the setup equal between our hosted service and on-prem. When updating, schema changes goes first and when successful the new version is allowed to be executed by users. We have small launcher app that the users run which starts the actual application to handle this. Our DB changes are always backwards compatible in the sense that the old version should operate correctly on an updated DB. Incompatible changes like renaming a column we handle by rolling out the new column first, then wait till the version still using the old column is retired before removing the column from the schema, or similar strategies. This way partial updates are safe, as the old app will just continue to use the partially updated DB as normal. We're migrating to a web app, and hope to have something similar. Plan is we'll update the tenant DB and once done we'll change the app routing so that new user sessions for that tenant goes to new app instances running the updated version. That said, we've not got 1 million users... yet.

SOLAR_FIELDS 2y ago

Good call out, it definitely reduces the blast radius to do it this way. Poor man’s isolation

deleted 2y ago
[deleted]

julik 2y ago

Good point. Also: nothing makes it impossible to apply migrations to multiple tenants in parallel, this is pretty much only IO. I didn't have to since my tenant numbers were very low, but very doable.

tyre 2y ago

Do you then run app instances per user? Or how do your upstream control for some DBs being on different schemas? At least with Single Point of Success database design you either move or don’t.

jasonthorsness 2y ago
App per tenant would be nice but in practice mostly I’ve seen shared app instances with a careful dance of “schema changes can’t break app” and “app can’t use new schema until it is 100% rolled out” which is super annoying.
- what 2y ago
  How is that super annoying? That’s exactly how migrations are supposed to be done, even without “app per tenant”.

hobs 2y ago

I think its ok if you have a traditional server approach, but in the cloud this is a really great way to make yourself pay a lot for a little. Most systems have a minimum amount of traffic they'll let you pay for, most object stores are not setup to stream small updates in a transactionally consistent way, there's a ton of complexity "in the small" as it were.

stackskipton 2y ago
Depends on the cloud and their model. Azure/Google have MySQL/PostGres options where you get a server they manage OS/Database software for but you can run as many databases as hardware will allow. Some of other cloud databases don't charge you for database but simply for usage so in that case, usage = customer revenue so cost should scale as you do.

netghost 2y ago

I'm curious, how do you handle being in a partial state? It seems like it could be operationally tricky to have some users migrated, some pending, and some in an error state. On the other hand, I could imagine doing lazy migrations as user or tenants sign in as a nice way to smooth out the increased effort in migrating if you can engineer for it.

njs12345 2y ago
When not per tenant you still have this problem, and typically resolve by releasing database migrations separately to the code that uses changes (ensuring backwards compatibility). I guess per-tenant you would handle similarly but build some tooling to monitor the state of migrations. It might even be easier in some ways, as you might be more able to take table locks when migrating per-tenant.
magicalhippo 2y ago
We've got a desktop B2B application which we also run with per-tenant DB. This makes the setup equal between our hosted service and on-prem. When updating, schema changes goes first and when successful the new version is allowed to be executed by users. We have small launcher app that the users run which starts the actual application to handle this. Our DB changes are always backwards compatible in the sense that the old version should operate correctly on an updated DB. Incompatible changes like renaming a column we handle by rolling out the new column first, then wait till the version still using the old column is retired before removing the column from the schema, or similar strategies. This way partial updates are safe, as the old app will just continue to use the partially updated DB as normal. We're migrating to a web app, and hope to have something similar. Plan is we'll update the tenant DB and once done we'll change the app routing so that new user sessions for that tenant goes to new app instances running the updated version. That said, we've not got 1 million users... yet.