4 ms·
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a ro
by lgrthmsprs 2mo ago
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
- locknitpicker 2mo ago> That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. T I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software. Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature. Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved. In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
- otterley 2mo agoBoth Google and Amazon do rollbacks instead of feature flag switches. I’m pretty sure they have some experience maintaining user-facing software.
- locknitpicker 2mo ago> Both Google and Amazon do rollbacks instead of feature flag switches. I'm sorry, but you're completely wrong. I know for a fact that Amazon uses extensively a A/B testing system for feature flags. Others in this discussion already mentioned it.
- otterley 2mo agoI’m not wrong. They do use feature flags, but not as a substitute for rolling back failed or buggy deployments. They (and Google too) use feature flags in the ways according to the best practices discussed in the article. Google also covers this well in their highly-regarded Site Reliability Engineering book. Source: worked at AWS for 7 years. And knock it off with the attitude.