3 ms·

Another thing about tech debt to watch out for is reusing previous flags left before your time [0]. [0] - https://en.wikipedia.org/wiki/Knight_Capital_Group#20

Another thing about tech debt to watch out for is reusing previous flags left before your time [0]. [0] - https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stock_trading_disruption https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...

hinkley 6y ago
I always push back or ignore people who complain about how you shouldn’t have to “burn in” new code. That if it’s not ready to go you shouldn’t commit the change at all. Knight Capital would likely not have lost hundreds of millions of dollars if someone had deleted that flag months before they needed the slot again. Someone in that time would have likely noticed the extra server that was running really old code. Flags defaulted to off on first deploy are, IMO, just another form of refactoring. Prove that you can do nothing before you prove you can do something.
- yojo 6y ago
  To be clear, I agree that it often makes sense for the code to be “off” on initial deploy. By “default”, I mean what does the code do in absence of feature flag data? Put another way, my specific concern is “what happens when the flag system fails?” If you accidentally drop your feature flag database, do all your features turn off, or do they all turn on? Superficially it might seem safer for them all to turn off in this failure mode. After all, flags are for experiments, and what’s wrong with disabling experiments? The problem is that companies I have worked for (and those of friends I talk to) do not purge 100% of flags corresponding to launched features. Once code has been built on top of these unpurged flags, bugs are almost guaranteed if they get turned off via system error.
  - hinkley 6y ago
    One the system I’m most familiar with, the system won’t start. But as it’s sourcing it’s data from a Raft protocol data store, that server wasn’t going to start anyway. The source of that data could fall behind though. So yes, you do want to generally be careful and occasionally use an antonym of the most obvious flag to guarantee that true is on and all other states mean off.

hinkley 6y ago

I always push back or ignore people who complain about how you shouldn’t have to “burn in” new code. That if it’s not ready to go you shouldn’t commit the change at all. Knight Capital would likely not have lost hundreds of millions of dollars if someone had deleted that flag months before they needed the slot again. Someone in that time would have likely noticed the extra server that was running really old code. Flags defaulted to off on first deploy are, IMO, just another form of refactoring. Prove that you can do nothing before you prove you can do something.

yojo 6y ago
To be clear, I agree that it often makes sense for the code to be “off” on initial deploy. By “default”, I mean what does the code do in absence of feature flag data? Put another way, my specific concern is “what happens when the flag system fails?” If you accidentally drop your feature flag database, do all your features turn off, or do they all turn on? Superficially it might seem safer for them all to turn off in this failure mode. After all, flags are for experiments, and what’s wrong with disabling experiments? The problem is that companies I have worked for (and those of friends I talk to) do not purge 100% of flags corresponding to launched features. Once code has been built on top of these unpurged flags, bugs are almost guaranteed if they get turned off via system error.
- hinkley 6y ago
  One the system I’m most familiar with, the system won’t start. But as it’s sourcing it’s data from a Raft protocol data store, that server wasn’t going to start anyway. The source of that data could fall behind though. So yes, you do want to generally be careful and occasionally use an antonym of the most obvious flag to guarantee that true is on and all other states mean off.