4 ms·

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If y

by kgeist 5y ago

>That said features and bug fixes were often times gated by feature flags Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy, then feature flags can't prevent badly tested migrations from corrupting the DB, locking tables and other sorts of regressions. If you run your migrations manually each time, then there's a chance that someone enables a feature toggle without running the required migrations, which can result in all sorts of downtime. Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late?

drewcoo 5y ago
> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the product better for customers. > another concern Avoiding "feature flag hell" is a valid concern. It has to be managed. The big problem with conflict is underlying tightly coupled code, though. That should be fixed. Note this is also solved by breaking up monoliths. > tight release schedule If a release in this sense is something product-led, then feature flags almost create an API boundary (a good thing!) between product and dev. Product can determine when their release (meaning set of feature flags to be flipped) is ready and ideally toggle themselves instead of roping devs into release management roles.
- cmeacham98 5y ago
  I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.
  - laurent123456 5y ago
    I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.
    - derefr 5y ago
      > Otherwise it wouldn't be possible to have migrations that work with both old and new code. Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction: 1. rename the original table "out of the way" of the old code 2. add a new column of the new type 3. run an "INSERT ... SELECT ..." to populate the new column from a transformation of existing data 4. drop the old column of the old type 5. rename the new column to the old column's name 6. define a view with the name of the original table, that just queries through to the new (original + renamed + modified) table for most of the original columns, but which continues to serve the no-longer-existing column with its previous value, by computing its old-type value from its new-type value (+ data in other columns, if necessary.) Then either make sure that the new code is reading directly from the new table; or create a trivial passthrough view for the new version to use as well. (IMHO, as long as you've got writable-view support, every application-visible "table" should really just be a view, with its name suffixed with the ABI-major-compatibility-version of the application using it. Then the infrastructure team — and more specifically, a DBA, if you've got one — can do whatever they like with the underlying tables: refactoring them, partitioning them, moving them to other shards and forwarding them, etc. As long as all the views still work, and still produce the same query results, it doesn't matter what's underneath them.)
      - freedomben 5y ago
        I wrote a blog about this for anyone who would like to learn more. The query strings get you around the paywall if it comes up: https://freedomben.medium.com/the-rules-of-clean-and-mostly-painless-migrations-75b8b0dec194?sk=789b59e1ce45077303daea7866acf508 https://freedomben.medium.com/the-rules-of-clean-and-mostly-... If anyone doesn't know what migrations are: https://freedomben.medium.com/what-are-database-migrations-5ad203d7d955?sk=c02e997ac674b3dc54f2e3f70d7e64f6 https://freedomben.medium.com/what-are-database-migrations-5...
- kgeist 5y ago
  >The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev environment but in production it would lead to timeouts in requests for the duration of the migration for large clients (our application is sharded per tenant) because the table was very large for some of them and the migration locked it. The staging environment helped us find the problem before hitting production because we routinely clone production data (deanonymized) of the largest tenants to find problems like this. It's not practical (and maybe not very legal too) to force every developer have an up-to-date copy of that database on every VM/laptop, and load tests in an environment very similar to production show more meaningful results overall. And feature flags wouldn't help either because they only guard code. So far I'm unconvinced, it sounds pretty risky to me to go straight to prod. I agree however that the concern about conflicts between feature toggles is largely a monolith problem, it's a communication problem when many teams make changes to the same codebase and are unaware of what the other teams are doing.
  - nicoburns 5y ago
    > Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. This is definitely best practice, but it's not strictly necessary if a small amount of downtime is acceptable. We only have customers in one timezone and minimal traffic overnight, so we have quite a lot of leeway with this. Frankly even during business hours small amounts of downtime (e.g. 5 minutes) would be well tolerated: it's a lot better than most of the other services they are used to using anyway.
  - withinboredom 5y ago
    > Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. This doesn't have to be true. You can create an entirely separate table with the new data. New code knows how to join on this table, old code doesn't and thus ignores the new data. It doesn't work for every kind of migration, but in my experience, it's preferred by some DBAs if you have billions and billions of rows. Example: `select user_id, coalesce(new_col2, old_col2) as maybe_new_data, new_col3 as new_data from old_table left join new_table using user_id limit 1`
toast0 5y ago
> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lot of frontends and a lot of sharded databases, you don't have much choice; if your schema is changing, you've got to have a multiphased push: a) push frontend that can deal with either schema b) migrate schema c) push frontend that uses new schema for new feature (with the understanding that the old frontend code will be running on some nodes) --- this part could be feature flagged d) data cleanup if necessary e) push code that can safely assume all frontends are new feature aware and all rows are new feature ready IMHO, this multiphase push is really needed regardless of scale, but if you're small, you can cross your fingers and hope. Or if you're willing to take downtime, you can bring down the service, make the database changes without concurrent access, and bring the service back with code assuming the changes; most people don't like downtime though.
- kgeist 5y ago
  >Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards have migrated. So there's a large window (sometimes up to an hour) when some shards see the new schema and others see the old schema (while still running old code). It's one click of a button, however, and one logical release, we don't split it into separate releases (so I view them as "automatic"). The problem still stays, though, that you can only guard code with feature flags, migrations can't be conditionally disabled. With this setup, if a poorly tested migration goes awry, it's even more difficult to rollback, because it will take another hour to roll back all the shards.
  - withinboredom 5y ago
    We don't have a staging environment (for the backend) at work either. However, depending on the size of the tables in-question, a migration might take days. Thus, we usually ask DBA's for a migration days/weeks before any code goes live. There's usually quite a bit of discussion, and sometimes suggestions for an entirely different table with a join and/or application-only (in code, multiple query) join.
bcherny 5y ago
How DB migrations work in particular (other things, like business logic, work similarly): https://news.ycombinator.com/item?id=29046303 https://news.ycombinator.com/item?id=29046303
MrTortoise 5y ago
what if you insisted that database down time for migrations was not acceptable and that your code needs to work with different versions of database (this can be adapters)?
pluies 5y ago
> Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late? Reminiscent of Knight Capital losing $440 million in 45 minutes via feature flags: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ https://dougseven.com/2014/04/17/knightmare-a-devops-caution...

by kgeist 5y ago

drewcoo 5y ago

> how do feature flags work with migrations? The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version. Part of the reason people started breaking up monoliths is that continuous deployment with a db-backed monolith can be brittle. And making it work well requires a whole bunch of brain power that could go into things like making the product better for customers. > another concern Avoiding "feature flag hell" is a valid concern. It has to be managed. The big problem with conflict is underlying tightly coupled code, though. That should be fixed. Note this is also solved by breaking up monoliths. > tight release schedule If a release in this sense is something product-led, then feature flags almost create an API boundary (a good thing!) between product and dev. Product can determine when their release (meaning set of feature flags to be flipped) is ready and ideally toggle themselves instead of roping devs into release management roles.

cmeacham98 5y ago
I think their question was more "if I wrote a migration that accidentally drops the users table, how does your system prevent that from running on production"? That's a pretty extreme case, but the tldr is how are you testing migrations if you don't have a staging environment.
- laurent123456 5y ago
  I'd think they create "append-only" migrations, that can only add columns or tables. Otherwise it wouldn't be possible to have migrations that work with both old and new code.
  - derefr 5y ago
    > Otherwise it wouldn't be possible to have migrations that work with both old and new code. Sure you can. Say that you've changed the type of a column in an incompatible way. You can, within a migration that executes as an SQL transaction: 1. rename the original table "out of the way" of the old code 2. add a new column of the new type 3. run an "INSERT ... SELECT ..." to populate the new column from a transformation of existing data 4. drop the old column of the old type 5. rename the new column to the old column's name 6. define a view with the name of the original table, that just queries through to the new (original + renamed + modified) table for most of the original columns, but which continues to serve the no-longer-existing column with its previous value, by computing its old-type value from its new-type value (+ data in other columns, if necessary.) Then either make sure that the new code is reading directly from the new table; or create a trivial passthrough view for the new version to use as well. (IMHO, as long as you've got writable-view support, every application-visible "table" should really just be a view, with its name suffixed with the ABI-major-compatibility-version of the application using it. Then the infrastructure team — and more specifically, a DBA, if you've got one — can do whatever they like with the underlying tables: refactoring them, partitioning them, moving them to other shards and forwarding them, etc. As long as all the views still work, and still produce the same query results, it doesn't matter what's underneath them.)
    - freedomben 5y ago
      I wrote a blog about this for anyone who would like to learn more. The query strings get you around the paywall if it comes up: https://freedomben.medium.com/the-rules-of-clean-and-mostly-painless-migrations-75b8b0dec194?sk=789b59e1ce45077303daea7866acf508 https://freedomben.medium.com/the-rules-of-clean-and-mostly-... If anyone doesn't know what migrations are: https://freedomben.medium.com/what-are-database-migrations-5ad203d7d955?sk=c02e997ac674b3dc54f2e3f70d7e64f6 https://freedomben.medium.com/what-are-database-migrations-5...
kgeist 5y ago
>The idea is to have migrations that are backward compatible so that the current version of your code can use the db and so can the new version Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. As an example of what I'm talking about: a few months ago we had a migration that passed all code reviews and worked great in the dev environment but in production it would lead to timeouts in requests for the duration of the migration for large clients (our application is sharded per tenant) because the table was very large for some of them and the migration locked it. The staging environment helped us find the problem before hitting production because we routinely clone production data (deanonymized) of the largest tenants to find problems like this. It's not practical (and maybe not very legal too) to force every developer have an up-to-date copy of that database on every VM/laptop, and load tests in an environment very similar to production show more meaningful results overall. And feature flags wouldn't help either because they only guard code. So far I'm unconvinced, it sounds pretty risky to me to go straight to prod. I agree however that the concern about conflicts between feature toggles is largely a monolith problem, it's a communication problem when many teams make changes to the same codebase and are unaware of what the other teams are doing.
- nicoburns 5y ago
  > Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. This is definitely best practice, but it's not strictly necessary if a small amount of downtime is acceptable. We only have customers in one timezone and minimal traffic overnight, so we have quite a lot of leeway with this. Frankly even during business hours small amounts of downtime (e.g. 5 minutes) would be well tolerated: it's a lot better than most of the other services they are used to using anyway.
- withinboredom 5y ago
  > Well, any migration has to be backward-compatible with the old code because old code is still running when a migration is taking place. This doesn't have to be true. You can create an entirely separate table with the new data. New code knows how to join on this table, old code doesn't and thus ignores the new data. It doesn't work for every kind of migration, but in my experience, it's preferred by some DBAs if you have billions and billions of rows. Example: `select user_id, coalesce(new_col2, old_col2) as maybe_new_data, new_col3 as new_data from old_table left join new_table using user_id limit 1`

toast0 5y ago

> Sorry for maybe a silly question, but how do feature flags work with migrations? If your migrations run automatically on deploy Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. I didn't work on FB databases while I was employed by them, but when you've got a lot of frontends and a lot of sharded databases, you don't have much choice; if your schema is changing, you've got to have a multiphased push: a) push frontend that can deal with either schema b) migrate schema c) push frontend that uses new schema for new feature (with the understanding that the old frontend code will be running on some nodes) --- this part could be feature flagged d) data cleanup if necessary e) push code that can safely assume all frontends are new feature aware and all rows are new feature ready IMHO, this multiphase push is really needed regardless of scale, but if you're small, you can cross your fingers and hope. Or if you're willing to take downtime, you can bring down the service, make the database changes without concurrent access, and bring the service back with code assuming the changes; most people don't like downtime though.

kgeist 5y ago
>Basically they don't. Database migration based on frontend deploy doesn't really make sense at facebook scale, because deploy is no where close to synchronous; even feature flag changes aren't synchronous. Our deployments aren't strictly "synchronous" either. We have thousands of database shards which are all migrated one by one (with some degree of parallelism), and new code is deployed only after all the shards have migrated. So there's a large window (sometimes up to an hour) when some shards see the new schema and others see the old schema (while still running old code). It's one click of a button, however, and one logical release, we don't split it into separate releases (so I view them as "automatic"). The problem still stays, though, that you can only guard code with feature flags, migrations can't be conditionally disabled. With this setup, if a poorly tested migration goes awry, it's even more difficult to rollback, because it will take another hour to roll back all the shards.
- withinboredom 5y ago
  We don't have a staging environment (for the backend) at work either. However, depending on the size of the tables in-question, a migration might take days. Thus, we usually ask DBA's for a migration days/weeks before any code goes live. There's usually quite a bit of discussion, and sometimes suggestions for an entirely different table with a join and/or application-only (in code, multiple query) join.

bcherny 5y ago

How DB migrations work in particular (other things, like business logic, work similarly): https://news.ycombinator.com/item?id=29046303 https://news.ycombinator.com/item?id=29046303

MrTortoise 5y ago

what if you insisted that database down time for migrations was not acceptable and that your code needs to work with different versions of database (this can be adapters)?

pluies 5y ago

> Another concern I have is that if a feature toggle isn't enabled in production for a long time (for us, several days is already a long time due to a tight release schedule) new changes to the codebase by another team can conflict with the disabled feature and, since it's disabled, you probably won't know there's a problem until it's too late? Reminiscent of Knight Capital losing $440 million in 45 minutes via feature flags: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ https://dougseven.com/2014/04/17/knightmare-a-devops-caution...