5 ms·

Worked for a bigtech well known name, large and extremely important project, literally the core of a service serving an enormous number of users, and feature fl

by drclau 5y ago

Worked for a bigtech well known name, large and extremely important project, literally the core of a service serving an enormous number of users, and feature flags were mandatory, no exceptions. I can't imagine working without feature flags. Being able to enable new features in particular deployment rings (canary, dogfood, various production rings or regions), or per users / user groups, enabling gradually (percentage) and so on, is invaluable. I really can't overstate this. Heck, we went as far as using feature flags for risky bugfixes even. We had also internal tools to easily work with and track feature flags. A downside is that although normally you'd want to remove old feature flags that become obsolete, this hasn't been done very often. What I suggested and we started doing was to tag the feature flags with the name of the author and the date at which they were added, and the same for the config updates, and usually ticket number and title for both case. This did help with tracking obsolescence, but obviously there was still a need to plan and do the actual work. Automating this process further was out of the question, due to the high risks involved. Edit: added the last paragraph.

spookthesunset 5y ago
That is the one problem we’ve found with feature flags. It’s very easy to forget to gut the “old” parts when they are turned off. In many cases I’ve seen two or three year-old feature flags whose stale end still remains because the developer and / or team that did the branch never cleaned up. It isn’t usually malicious or lazy… it’s just how things would pan out. Then we’d be nervous removing the old stuff cause who knows why it was left there and who knows if they’d want it back on again…
- obstacle1 5y ago
  That's not a problem with feature flags per se, it's a problem with lazy implementations of feature flags. Flags should be associated with an expiry date, and company comms tooling should be consistently yelling in some public channel when expired flags still exist in the codebase.
  - spookthesunset 5y ago
    > That's not a problem with feature flags per se, it's a problem with lazy implementations of feature flags. Oh absolutely. Feature flags are great, but you definitely need discipline to make sure you clean things up. The longer the unused code rots, the harder it is to remove it.
- pkaeding 5y ago
  I have found it can be good to 'remove' the flag at the same time you create it, but just don't merge the removal until later. I wrote up this idea in a blog post a while ago, if anyone finds it interesting: https://launchdarkly.com/blog/how-to-use-feature-flags-without-technical-debt https://launchdarkly.com/blog/how-to-use-feature-flags-witho...
  - adamredwoods 5y ago
    I've seen this used, but as PRs get added, these 'cleanup' PRs move to the bottom and are usually ignored by other team members. To me, it's about having enough time to do this in a sprint, and that means it really needs to be a post-launch Jira ticket. Which I've seen done maybe once.
    - pkaeding 5y ago
      yeah, this is definitely a risk. I agree that the PR needs to be tracked as a post-launch task. The advantage to this approach is that you do the hard part of removing the flag (ie, thinking through all the parts of the code that need to be cleaned up) while everything is fresh in your mind. Otherwise, you end up spending more time regaining all of the context, and are more likely to leave some vestigial dead code because you aren't sure it isn't needed any more (this is probably less of a risk with languages that lend themselves well to static analysis that can identify dead code, but these tools are never perfect).
- ruh-roh 5y ago
  Starting new at a startup-with-traction several years ago, I had reason to check out the feature flag configs for my first feature. It was hilarious. - There were hundreds of them, some of them going back to the garage days, multiple years old. Some would turn on/off major functionality core to the product. For example, this was an ecommerce product - one toggle was "show/hide the buy button". (I think that one is staying in.) - This was all hand-rolled, pre-LaunchDarkly stuff, but at least they were all in one place. I diffed Production, Staging, and a one-off UAT environment - the toggles were MASSIVELY different across each.
- adamredwoods 5y ago
  Same. Feature flags need a yearly audit, but who has bandwidth for that? I'd file it under tech debt / tech health, or a great project for entry level or new hire.
bschne 5y ago
> A downside is that although normally you'd want to remove old feature flags that become obsolete, this hasn't been done very often. I figure this should be somewhat automatable since the relevant bits of code have references to the exact flag's identifier. Was it not done b/c nobody was incentivized in any way to spend any kind of time on cleanup? edit: OK that might be a bit naive on the organizational side. Force everyone to give every flag an expiry date for review or something maybe?
- yojo 5y ago
  A typical pattern I saw was a team adds a new flag, then gradually rolls it out. Once done the removal is added to the backlog. “High priority” bug fixes and feature work gradually moves the ticket down in the system. Two months later a reorg happens, the team no longer exists, and the work is lost.
- hosh 5y ago
  In the case of my team, although we recognize we should clean it up, we’re usually prioritizing pushing out more features. That tendency leads to a lot of old feature flags. In a startup, being able to get features out is more important than cleaning up old flags. If we don’t achieve product fit before we run out of runway, then the whole thing can get shut down. If we don’t achieve customer validation, same thing. Usually, at some point, someone will suggest a rewrite. That pretty much never goes well.
the_gipsy 5y ago
> What I suggested and we started doing was to tag the feature flags with the name of the author I suggest tagging the PO/manager who requested the feature.
bob1029 5y ago
> A downside is that although normally you'd want to remove old feature flags that become obsolete, this hasn't been done very often. We use feature flags a ton and this is something that burns us a little bit too. The upsides are worth it though. Having that surgical precision in production is essential when you have a complex product that touches many lines of business.
8note 5y ago
I turned off a giant old service that was full of feature flags once. Having them everywhere was great, we could flip things off feature by feature just by changing the flags. Having the old ones still around can be handy.
cammil 5y ago
Where did you store the flag? In code? App config? Or operational data store?

by drclau 5y ago

spookthesunset 5y ago

That is the one problem we’ve found with feature flags. It’s very easy to forget to gut the “old” parts when they are turned off. In many cases I’ve seen two or three year-old feature flags whose stale end still remains because the developer and / or team that did the branch never cleaned up. It isn’t usually malicious or lazy… it’s just how things would pan out. Then we’d be nervous removing the old stuff cause who knows why it was left there and who knows if they’d want it back on again…

obstacle1 5y ago
That's not a problem with feature flags per se, it's a problem with lazy implementations of feature flags. Flags should be associated with an expiry date, and company comms tooling should be consistently yelling in some public channel when expired flags still exist in the codebase.
- spookthesunset 5y ago
  > That's not a problem with feature flags per se, it's a problem with lazy implementations of feature flags. Oh absolutely. Feature flags are great, but you definitely need discipline to make sure you clean things up. The longer the unused code rots, the harder it is to remove it.
pkaeding 5y ago
I have found it can be good to 'remove' the flag at the same time you create it, but just don't merge the removal until later. I wrote up this idea in a blog post a while ago, if anyone finds it interesting: https://launchdarkly.com/blog/how-to-use-feature-flags-without-technical-debt https://launchdarkly.com/blog/how-to-use-feature-flags-witho...
- adamredwoods 5y ago
  I've seen this used, but as PRs get added, these 'cleanup' PRs move to the bottom and are usually ignored by other team members. To me, it's about having enough time to do this in a sprint, and that means it really needs to be a post-launch Jira ticket. Which I've seen done maybe once.
  - pkaeding 5y ago
    yeah, this is definitely a risk. I agree that the PR needs to be tracked as a post-launch task. The advantage to this approach is that you do the hard part of removing the flag (ie, thinking through all the parts of the code that need to be cleaned up) while everything is fresh in your mind. Otherwise, you end up spending more time regaining all of the context, and are more likely to leave some vestigial dead code because you aren't sure it isn't needed any more (this is probably less of a risk with languages that lend themselves well to static analysis that can identify dead code, but these tools are never perfect).
ruh-roh 5y ago
Starting new at a startup-with-traction several years ago, I had reason to check out the feature flag configs for my first feature. It was hilarious. - There were hundreds of them, some of them going back to the garage days, multiple years old. Some would turn on/off major functionality core to the product. For example, this was an ecommerce product - one toggle was "show/hide the buy button". (I think that one is staying in.) - This was all hand-rolled, pre-LaunchDarkly stuff, but at least they were all in one place. I diffed Production, Staging, and a one-off UAT environment - the toggles were MASSIVELY different across each.
adamredwoods 5y ago
Same. Feature flags need a yearly audit, but who has bandwidth for that? I'd file it under tech debt / tech health, or a great project for entry level or new hire.

bschne 5y ago

> A downside is that although normally you'd want to remove old feature flags that become obsolete, this hasn't been done very often. I figure this should be somewhat automatable since the relevant bits of code have references to the exact flag's identifier. Was it not done b/c nobody was incentivized in any way to spend any kind of time on cleanup? edit: OK that might be a bit naive on the organizational side. Force everyone to give every flag an expiry date for review or something maybe?

yojo 5y ago
A typical pattern I saw was a team adds a new flag, then gradually rolls it out. Once done the removal is added to the backlog. “High priority” bug fixes and feature work gradually moves the ticket down in the system. Two months later a reorg happens, the team no longer exists, and the work is lost.
hosh 5y ago
In the case of my team, although we recognize we should clean it up, we’re usually prioritizing pushing out more features. That tendency leads to a lot of old feature flags. In a startup, being able to get features out is more important than cleaning up old flags. If we don’t achieve product fit before we run out of runway, then the whole thing can get shut down. If we don’t achieve customer validation, same thing. Usually, at some point, someone will suggest a rewrite. That pretty much never goes well.

the_gipsy 5y ago

> What I suggested and we started doing was to tag the feature flags with the name of the author I suggest tagging the PO/manager who requested the feature.

bob1029 5y ago

> A downside is that although normally you'd want to remove old feature flags that become obsolete, this hasn't been done very often. We use feature flags a ton and this is something that burns us a little bit too. The upsides are worth it though. Having that surgical precision in production is essential when you have a complex product that touches many lines of business.

8note 5y ago

I turned off a giant old service that was full of feature flags once. Having them everywhere was great, we could flip things off feature by feature just by changing the flags. Having the old ones still around can be handy.

cammil 5y ago

Where did you store the flag? In code? App config? Or operational data store?