3 ms·

In many cases you are better off using a kill switch than a feature flag. This may seem pedantic, but the way your system fails (on vs off) can protect you from

by yojo 6y ago

In many cases you are better off using a kill switch than a feature flag. This may seem pedantic, but the way your system fails (on vs off) can protect you from disaster when your flag setting framework has a bug. On a large codebase it is easy to forget to clean these things up, and a flag that hasn’t been set to off in a year can be masking a major regression. At my last job we had two major outages in as many years from defunct flags defaulting to “off” when the feature flag system failed to return flag states. Failing to “on” is a simple design choice to protect you from your tech debt. There are more expensive better fixes (e.g. automated enforcement of removing flags from codebase), but none as easy to implement.

gravypod 6y ago
Another thing about tech debt to watch out for is reusing previous flags left before your time [0]. [0] - https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...
- hinkley 6y ago
  I always push back or ignore people who complain about how you shouldn’t have to “burn in” new code. That if it’s not ready to go you shouldn’t commit the change at all. Knight Capital would likely not have lost hundreds of millions of dollars if someone had deleted that flag months before they needed the slot again. Someone in that time would have likely noticed the extra server that was running really old code. Flags defaulted to off on first deploy are, IMO, just another form of refactoring. Prove that you can do nothing before you prove you can do something.
  - yojo 6y ago
    To be clear, I agree that it often makes sense for the code to be “off” on initial deploy. By “default”, I mean what does the code do in absence of feature flag data? Put another way, my specific concern is “what happens when the flag system fails?” If you accidentally drop your feature flag database, do all your features turn off, or do they all turn on? Superficially it might seem safer for them all to turn off in this failure mode. After all, flags are for experiments, and what’s wrong with disabling experiments? The problem is that companies I have worked for (and those of friends I talk to) do not purge 100% of flags corresponding to launched features. Once code has been built on top of these unpurged flags, bugs are almost guaranteed if they get turned off via system error.
    - hinkley 6y ago
      One the system I’m most familiar with, the system won’t start. But as it’s sourcing it’s data from a Raft protocol data store, that server wasn’t going to start anyway. The source of that data could fall behind though. So yes, you do want to generally be careful and occasionally use an antonym of the most obvious flag to guarantee that true is on and all other states mean off.
kqr 6y ago
I'm not sure I agree completely with this. Defaulting to "on" and only rarely going to "off" would, at least in the teams I've worked, result in the "off" path being much less tested and more likely to contain regressions. If, then, the "off" state is meant as an emergency oops-this-new-code-didnt-work-well-at-all-lets-revert-to-a-safe-state then having it be the less tested state seems like a problem waiting to happen. In other words, by defaulting to "on", you get two states that are likely to contain bugs: the on state, because it contains new code, and the off state, because it's the less rigorously exercised configuration. The nice thing about defaulting to off is that in that case, the off state will contain both old, known code, and be rigorously exercised. The on state will be the brittle one, but it would be anyway on account of running the new code. I guess we can both agree defaults may seem pedantic but matter a lot.
- yojo 6y ago
  In my experience the “off” path is only well tested the day the feature is released. Once the feature is out in the world the old tests go stale, and manual testing in effect never happens. It gets worse when a new ungated feature is built on top of something that was behind a flag (but always set to on). Suddenly code that was expected to always run doesn’t, and all kind of expectations get broken. I’m most concerned about “What happens when your flag manager fails to return data.” This should be a rare event. In my previous organization (~500 devs) it happened twice, and was catastrophic for the ~4 hours to get everything fixed. If instead some unpolished features had leaked for four hours, no one would have cared. If your organization has excellent hygiene at cleaning up flags, then off may be the sensible default, since you won’t have 10 years of cruft blowing up at once. But building a culture with that kind of discipline is hard, and automating it is non-trivial/often difficult to get developer resources for. Defaulting to “on” can be a pragmatic choice when you can’t invest in paying off all the tech debt.
  - kqr 6y ago
    That might very well be true. I'm fortunate enough to work in an organisation that takes tech debt somewhat seriously and we never jam a flag "on". If we're at the point where we're ready to turn it on for everyone, we just remove it from the code. It's either on for a subset of users, or it's not in there at all.
buster 6y ago
Usually you remove flags after they heva ebeen on for some time. I find in a "every few weeks" release cycle it works well to remove the flags of the last release (which by then have been live for a few weeks).

by yojo 6y ago

gravypod 6y ago

Another thing about tech debt to watch out for is reusing previous flags left before your time [0]. [0] - https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...

hinkley 6y ago
I always push back or ignore people who complain about how you shouldn’t have to “burn in” new code. That if it’s not ready to go you shouldn’t commit the change at all. Knight Capital would likely not have lost hundreds of millions of dollars if someone had deleted that flag months before they needed the slot again. Someone in that time would have likely noticed the extra server that was running really old code. Flags defaulted to off on first deploy are, IMO, just another form of refactoring. Prove that you can do nothing before you prove you can do something.
- yojo 6y ago
  To be clear, I agree that it often makes sense for the code to be “off” on initial deploy. By “default”, I mean what does the code do in absence of feature flag data? Put another way, my specific concern is “what happens when the flag system fails?” If you accidentally drop your feature flag database, do all your features turn off, or do they all turn on? Superficially it might seem safer for them all to turn off in this failure mode. After all, flags are for experiments, and what’s wrong with disabling experiments? The problem is that companies I have worked for (and those of friends I talk to) do not purge 100% of flags corresponding to launched features. Once code has been built on top of these unpurged flags, bugs are almost guaranteed if they get turned off via system error.
  - hinkley 6y ago
    One the system I’m most familiar with, the system won’t start. But as it’s sourcing it’s data from a Raft protocol data store, that server wasn’t going to start anyway. The source of that data could fall behind though. So yes, you do want to generally be careful and occasionally use an antonym of the most obvious flag to guarantee that true is on and all other states mean off.

kqr 6y ago

I'm not sure I agree completely with this. Defaulting to "on" and only rarely going to "off" would, at least in the teams I've worked, result in the "off" path being much less tested and more likely to contain regressions. If, then, the "off" state is meant as an emergency oops-this-new-code-didnt-work-well-at-all-lets-revert-to-a-safe-state then having it be the less tested state seems like a problem waiting to happen. In other words, by defaulting to "on", you get two states that are likely to contain bugs: the on state, because it contains new code, and the off state, because it's the less rigorously exercised configuration. The nice thing about defaulting to off is that in that case, the off state will contain both old, known code, and be rigorously exercised. The on state will be the brittle one, but it would be anyway on account of running the new code. I guess we can both agree defaults may seem pedantic but matter a lot.

yojo 6y ago
In my experience the “off” path is only well tested the day the feature is released. Once the feature is out in the world the old tests go stale, and manual testing in effect never happens. It gets worse when a new ungated feature is built on top of something that was behind a flag (but always set to on). Suddenly code that was expected to always run doesn’t, and all kind of expectations get broken. I’m most concerned about “What happens when your flag manager fails to return data.” This should be a rare event. In my previous organization (~500 devs) it happened twice, and was catastrophic for the ~4 hours to get everything fixed. If instead some unpolished features had leaked for four hours, no one would have cared. If your organization has excellent hygiene at cleaning up flags, then off may be the sensible default, since you won’t have 10 years of cruft blowing up at once. But building a culture with that kind of discipline is hard, and automating it is non-trivial/often difficult to get developer resources for. Defaulting to “on” can be a pragmatic choice when you can’t invest in paying off all the tech debt.
- kqr 6y ago
  That might very well be true. I'm fortunate enough to work in an organisation that takes tech debt somewhat seriously and we never jam a flag "on". If we're at the point where we're ready to turn it on for everyone, we just remove it from the code. It's either on for a subset of users, or it's not in there at all.

buster 6y ago

Usually you remove flags after they heva ebeen on for some time. I find in a "every few weeks" release cycle it works well to remove the flags of the last release (which by then have been live for a few weeks).