5 ms·

This isn't a perfect solution but we "solved" this by allowing services to know if migrations have been run. We store the migration data in a shared database an

This isn't a perfect solution but we "solved" this by allowing services to know if migrations have been run. We store the migration data in a shared database and every service that depends on a specific migration, directly or indirectly, is configured to verify that migration has been run before health checks will pass. The system is designed to support automatic migrations and deployments, but I don't trust it enough yet. (It's easy to write a migration that works on a local database but consumes too many resources in production, so it's just a matter of having a proper preprod, that I haven't created yet.)

bredren 5y ago
How do you sync the state of production feature flag toggles, and FF config for developer environment or for standing up a new developer environment? For example, a new full-stack SWE is onboarding and has an env representing production, populated with test data. That engineer wants to run the product as it currently exists on production including the state of feature flags. How does this developer update this state, or remain notified if a FF state changes that affects something they are working on?
- notreallyserio 5y ago
  We don't really use feature flags, this migration code is just our way to get closer. But it would be possible to write a migration that handles the feature flag swaps, assuming it's a value in a database or something that works somewhat like a database (redis, storage bucket, k8s configmap, etc.) That's how I would do it anyway. The migration content is all committed to a central repository (we're using a monorepo) and they can basically run "migrate all" to get them. There's also a script that creates a basic environment from scratch, but it comes with only minimal test data, and not nearly enough for load testing. The dev would need to watch for updates to the main branch to know if something there requires their attention, which may not be totally scalable. We're miniscule and pre-launch so it's not a big deal for us yet.
- dpark 5y ago
  Build a service/loop/CI/cron job/whatever to pull feature flag state from prod and check into your relevant branch(es). Make the dev deployment apply this state by default. You’ll get a lot of extra value out of this if you have an automated test pass that exercises the bulk of the product. You get bonus points if you can integrate individual feature flags back to your branch(es) independently, because it will let you easily identify which flag has broken automation.
  - bredren 5y ago
    This makes sense, thanks for this set of ideas. Have you implemented this, or seen write-ups of this pattern described at length? Or perhaps commercial products providing for this scenario? Curious about prior art on this subject.
    - dpark 5y ago
      My team has basically implemented this twice, for two different products. One (product A) does individual integrations from the production feature flag system to the repo when the flag is at 100% in production. It’s quite successful. The other (product B) does bulk integration every few hours from production to the repo, bringing all flags at once. It enables flags that have been “approved” (as opposed to fully enabled in production). It’s less successful because of the bulk behavior, which makes it more difficult to isolate breaks. Both of these integrations go through the same validation gates as pull requests (they are actually implemented as pull requests). This ensures that flag changes through this system cannot break pull requests. The ideal system for protecting pull requests (or whatever other validation) is to require validation to pass before the flag can be enabled in production. I have not implemented this as a hard gate, but I do have an “fyi” validation run as post of the flag enabling for product A so engineers see if they’ve broken something before they start turning on the flag. Getting philosophical, I strongly believe that feature flags should not be enabled by default in the validation environment until they are fully enabled (or close to fully enabled) in production. This ensures that the base state (flag off) is validated. You always want it to be safe to turn off the flag, which means you want validation running with the flag off. Product B does not have this behavior for legacy reasons but I think it’s a poor technical decision.

bredren 5y ago

How do you sync the state of production feature flag toggles, and FF config for developer environment or for standing up a new developer environment? For example, a new full-stack SWE is onboarding and has an env representing production, populated with test data. That engineer wants to run the product as it currently exists on production including the state of feature flags. How does this developer update this state, or remain notified if a FF state changes that affects something they are working on?

notreallyserio 5y ago
We don't really use feature flags, this migration code is just our way to get closer. But it would be possible to write a migration that handles the feature flag swaps, assuming it's a value in a database or something that works somewhat like a database (redis, storage bucket, k8s configmap, etc.) That's how I would do it anyway. The migration content is all committed to a central repository (we're using a monorepo) and they can basically run "migrate all" to get them. There's also a script that creates a basic environment from scratch, but it comes with only minimal test data, and not nearly enough for load testing. The dev would need to watch for updates to the main branch to know if something there requires their attention, which may not be totally scalable. We're miniscule and pre-launch so it's not a big deal for us yet.
dpark 5y ago
Build a service/loop/CI/cron job/whatever to pull feature flag state from prod and check into your relevant branch(es). Make the dev deployment apply this state by default. You’ll get a lot of extra value out of this if you have an automated test pass that exercises the bulk of the product. You get bonus points if you can integrate individual feature flags back to your branch(es) independently, because it will let you easily identify which flag has broken automation.
- bredren 5y ago
  This makes sense, thanks for this set of ideas. Have you implemented this, or seen write-ups of this pattern described at length? Or perhaps commercial products providing for this scenario? Curious about prior art on this subject.
  - dpark 5y ago
    My team has basically implemented this twice, for two different products. One (product A) does individual integrations from the production feature flag system to the repo when the flag is at 100% in production. It’s quite successful. The other (product B) does bulk integration every few hours from production to the repo, bringing all flags at once. It enables flags that have been “approved” (as opposed to fully enabled in production). It’s less successful because of the bulk behavior, which makes it more difficult to isolate breaks. Both of these integrations go through the same validation gates as pull requests (they are actually implemented as pull requests). This ensures that flag changes through this system cannot break pull requests. The ideal system for protecting pull requests (or whatever other validation) is to require validation to pass before the flag can be enabled in production. I have not implemented this as a hard gate, but I do have an “fyi” validation run as post of the flag enabling for product A so engineers see if they’ve broken something before they start turning on the flag. Getting philosophical, I strongly believe that feature flags should not be enabled by default in the validation environment until they are fully enabled (or close to fully enabled) in production. This ensures that the base state (flag off) is validated. You always want it to be safe to turn off the flag, which means you want validation running with the flag off. Product B does not have this behavior for legacy reasons but I think it’s a poor technical decision.