4 ms·

I know you said frontend, but we use them for both FE and BE. They work fine, but our problem comes with DB migrations. We just haven't found a good way to deal

I know you said frontend, but we use them for both FE and BE. They work fine, but our problem comes with DB migrations. We just haven't found a good way to deal with DB changes and flags.

beberlei 5y ago
It helps if you follow these rules: Never modify a column Never drop a column or table
notreallyserio 5y ago
This isn't a perfect solution but we "solved" this by allowing services to know if migrations have been run. We store the migration data in a shared database and every service that depends on a specific migration, directly or indirectly, is configured to verify that migration has been run before health checks will pass. The system is designed to support automatic migrations and deployments, but I don't trust it enough yet. (It's easy to write a migration that works on a local database but consumes too many resources in production, so it's just a matter of having a proper preprod, that I haven't created yet.)
- bredren 5y ago
  How do you sync the state of production feature flag toggles, and FF config for developer environment or for standing up a new developer environment? For example, a new full-stack SWE is onboarding and has an env representing production, populated with test data. That engineer wants to run the product as it currently exists on production including the state of feature flags. How does this developer update this state, or remain notified if a FF state changes that affects something they are working on?
  - notreallyserio 5y ago
    We don't really use feature flags, this migration code is just our way to get closer. But it would be possible to write a migration that handles the feature flag swaps, assuming it's a value in a database or something that works somewhat like a database (redis, storage bucket, k8s configmap, etc.) That's how I would do it anyway. The migration content is all committed to a central repository (we're using a monorepo) and they can basically run "migrate all" to get them. There's also a script that creates a basic environment from scratch, but it comes with only minimal test data, and not nearly enough for load testing. The dev would need to watch for updates to the main branch to know if something there requires their attention, which may not be totally scalable. We're miniscule and pre-launch so it's not a big deal for us yet.
  - dpark 5y ago
    Build a service/loop/CI/cron job/whatever to pull feature flag state from prod and check into your relevant branch(es). Make the dev deployment apply this state by default. You’ll get a lot of extra value out of this if you have an automated test pass that exercises the bulk of the product. You get bonus points if you can integrate individual feature flags back to your branch(es) independently, because it will let you easily identify which flag has broken automation.
    - bredren 5y ago
      This makes sense, thanks for this set of ideas. Have you implemented this, or seen write-ups of this pattern described at length? Or perhaps commercial products providing for this scenario? Curious about prior art on this subject.
      - dpark 5y ago
        My team has basically implemented this twice, for two different products. One (product A) does individual integrations from the production feature flag system to the repo when the flag is at 100% in production. It’s quite successful. The other (product B) does bulk integration every few hours from production to the repo, bringing all flags at once. It enables flags that have been “approved” (as opposed to fully enabled in production). It’s less successful because of the bulk behavior, which makes it more difficult to isolate breaks. Both of these integrations go through the same validation gates as pull requests (they are actually implemented as pull requests). This ensures that flag changes through this system cannot break pull requests. The ideal system for protecting pull requests (or whatever other validation) is to require validation to pass before the flag can be enabled in production. I have not implemented this as a hard gate, but I do have an “fyi” validation run as post of the flag enabling for product A so engineers see if they’ve broken something before they start turning on the flag. Getting philosophical, I strongly believe that feature flags should not be enabled by default in the validation environment until they are fully enabled (or close to fully enabled) in production. This ensures that the base state (flag off) is validated. You always want it to be safe to turn off the flag, which means you want validation running with the flag off. Product B does not have this behavior for legacy reasons but I think it’s a poor technical decision.
deleted 5y ago
[deleted]
dpark 5y ago
> our problem comes with DB migrations What kind of problems do you run into? And how is your DB deployed/updated/migrated/whatever? I’ve found it necessary to either always update the DB before the code/binaries or to expose DB version to the code so that it can automatically turn off features if the DB schema isn’t updated yet. Which of these options makes sense depends on how you deploy. If you can rigidly enforce that the DB always updates before binaries, that’s a simpler model (but your DB queries need to be backwards/forwards compatible, depending on whether they are in code or in stored procedures). The other thing that’s been hugely successful is to have a comprehensive test suite that can be exercised in intermediate states. e.g. New DB with old binaries. Absent a strong test suite, it’s always human judgement when it’s safe to roll out. I’ve also found success with taking certain actions out of the deployment entirely. e.g. Adding indexes to large tables gets done independent of deployments, because if things go south we don’t want to be trying to resolve the indexing issue in the middle of a deployment. It’s easier to do this in isolation from other changes.

I know you said frontend, but we use them for both FE and BE. They work fine, but our problem comes with DB migrations. We just haven't found a good way to deal with DB changes and flags.

beberlei 5y ago

It helps if you follow these rules: Never modify a column Never drop a column or table

notreallyserio 5y ago

This isn't a perfect solution but we "solved" this by allowing services to know if migrations have been run. We store the migration data in a shared database and every service that depends on a specific migration, directly or indirectly, is configured to verify that migration has been run before health checks will pass. The system is designed to support automatic migrations and deployments, but I don't trust it enough yet. (It's easy to write a migration that works on a local database but consumes too many resources in production, so it's just a matter of having a proper preprod, that I haven't created yet.)

bredren 5y ago
How do you sync the state of production feature flag toggles, and FF config for developer environment or for standing up a new developer environment? For example, a new full-stack SWE is onboarding and has an env representing production, populated with test data. That engineer wants to run the product as it currently exists on production including the state of feature flags. How does this developer update this state, or remain notified if a FF state changes that affects something they are working on?
- notreallyserio 5y ago
  We don't really use feature flags, this migration code is just our way to get closer. But it would be possible to write a migration that handles the feature flag swaps, assuming it's a value in a database or something that works somewhat like a database (redis, storage bucket, k8s configmap, etc.) That's how I would do it anyway. The migration content is all committed to a central repository (we're using a monorepo) and they can basically run "migrate all" to get them. There's also a script that creates a basic environment from scratch, but it comes with only minimal test data, and not nearly enough for load testing. The dev would need to watch for updates to the main branch to know if something there requires their attention, which may not be totally scalable. We're miniscule and pre-launch so it's not a big deal for us yet.
- dpark 5y ago
  Build a service/loop/CI/cron job/whatever to pull feature flag state from prod and check into your relevant branch(es). Make the dev deployment apply this state by default. You’ll get a lot of extra value out of this if you have an automated test pass that exercises the bulk of the product. You get bonus points if you can integrate individual feature flags back to your branch(es) independently, because it will let you easily identify which flag has broken automation.
  - bredren 5y ago
    This makes sense, thanks for this set of ideas. Have you implemented this, or seen write-ups of this pattern described at length? Or perhaps commercial products providing for this scenario? Curious about prior art on this subject.
    - dpark 5y ago
      My team has basically implemented this twice, for two different products. One (product A) does individual integrations from the production feature flag system to the repo when the flag is at 100% in production. It’s quite successful. The other (product B) does bulk integration every few hours from production to the repo, bringing all flags at once. It enables flags that have been “approved” (as opposed to fully enabled in production). It’s less successful because of the bulk behavior, which makes it more difficult to isolate breaks. Both of these integrations go through the same validation gates as pull requests (they are actually implemented as pull requests). This ensures that flag changes through this system cannot break pull requests. The ideal system for protecting pull requests (or whatever other validation) is to require validation to pass before the flag can be enabled in production. I have not implemented this as a hard gate, but I do have an “fyi” validation run as post of the flag enabling for product A so engineers see if they’ve broken something before they start turning on the flag. Getting philosophical, I strongly believe that feature flags should not be enabled by default in the validation environment until they are fully enabled (or close to fully enabled) in production. This ensures that the base state (flag off) is validated. You always want it to be safe to turn off the flag, which means you want validation running with the flag off. Product B does not have this behavior for legacy reasons but I think it’s a poor technical decision.

deleted 5y ago

[deleted]

dpark 5y ago

> our problem comes with DB migrations What kind of problems do you run into? And how is your DB deployed/updated/migrated/whatever? I’ve found it necessary to either always update the DB before the code/binaries or to expose DB version to the code so that it can automatically turn off features if the DB schema isn’t updated yet. Which of these options makes sense depends on how you deploy. If you can rigidly enforce that the DB always updates before binaries, that’s a simpler model (but your DB queries need to be backwards/forwards compatible, depending on whether they are in code or in stored procedures). The other thing that’s been hugely successful is to have a comprehensive test suite that can be exercised in intermediate states. e.g. New DB with old binaries. Absent a strong test suite, it’s always human judgement when it’s safe to roll out. I’ve also found success with taking certain actions out of the deployment entirely. e.g. Adding indexes to large tables gets done independent of deployments, because if things go south we don’t want to be trying to resolve the indexing issue in the middle of a deployment. It’s easier to do this in isolation from other changes.