4 ms·

>>"With feature flags, I can safely test in production without fear of breaking something or negatively affecting the customer experience." That's not true at

by awill 6y ago

>>"With feature flags, I can safely test in production without fear of breaking something or negatively affecting the customer experience." That's not true at all. It is a common misconception that feature flags have no risk.

aussiedude 6y ago
Agree 100%. Just worked on a incident last week which was caused by feature flags.
- NickNameNick 6y ago
  The Knight Capital bankruptcy was caused in part by an error related to a feature flag, Which cost them ~US$440 million in trading errors in ~45 minutes.
necovek 6y ago
I would actually venture to say that the way they are usually perused, they increase the risk! Common approach is to just plaster the codebase with "if feature_flag(x)", so you complicate dozens of files/modules/functions, make them harder to reason about and debug. And cleaning it up is commonly a mess. Another approach is to duplicate code which frequently leaves one branch untested. FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master. And even then the complexity they introduce is sometimes a burden, but overall it's worth it over alternative approaches. And sure, you can start implementing stuff iteratively, in small chunks and perusing feature flags to hide bits of unfinished user flow, but the benefits you are getting there are from the former (iterative, small changes), rather than from feature flags. It is still no substitution for testing in a prod-like environment, at the very least that you are not breaking any of the integration places which are not covered by tests (and integration and system tests are slow, which always puts a boundary on how many you can have).
- mnsc 6y ago
  >FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master But isn't this case what staging originally tried to solve? "this big feature isn't ready for production but we need to regression test in a setup as similar to production as possible" ?
  - necovek 6y ago
    Not that I know: if you needed a "staging" for each of the feature branches, the complexity and cost grew, but problems remained (i.e. merging trunk and a branch back and forth). For the last 15 years I've been at teams using a "staging", it was simply a single production-like environment (well, my first project was exactly like production on slightly less beefy hardware with sanitized production DB import nightly) that gets the "trunk" (or "main") deployed to automatically (on my current team, it can be used to run non-production code as part of the deployment process which you bail out of). I am pretty sure there is no one true definition of staging, so it's a spectrum rather than an exact way to do things.
    - mnsc 6y ago
      I'm talking about the prehistoric time before git and feature branches. Where all the developers hacked away for months on That New Feature and all code just went into trunk, cause that's where the development is. Back then staging was invaluable to get the confidence to do that (bi?)annual major release.
      - necovek 6y ago
        Sure, that's the time we learned why feature branches and not releasing often sucks :) And yes, you sometimes had "staging" track your trunk, when you would be missing a staging environment for production where you'd want to test critical fixes, so ultimately, you needed both then.
  - loopz 6y ago
    Staging has always been about QA before deployment in production. For tools you buy, one might just end up forgoing staging where you know it must work. Though, what to do if an update bricks the system? For any non-trivial and complex software, you still end up with having at least one test-environment though. Feature flags is about multiple branching. Staging doesn't really give you that in full, just by hacking the environment. It's then that you end up with environments that don't properly reflect future production and start missing stuff. The author sounds like someone familiar with testing, that needs to learn about CI/CD. Sure, for trivial front end stuff, you can often get away with doing it in production. But without synchronization between environments, you end up in trouble and take on more risk, just for the luxury of treating your paying customers as beta-testers. It's a valid opinion depending on what one is working on, but I bet it doesn't apply for most outside unimportant side-projects.
hinkley 6y ago
I suspect it’s the same quality of thought that can’t fathom that a test could have a bug. Implementing a flag takes fewer lines of code than the feature, and so while the odds are very high that the bug will be within a feature, there is a non zero chance that the bug is in the flag. A bug might break a new feature. It might break the old behavior. It may turn the system into a latch (you can launch it darkly, turn it on, and then not be able to turn it off). And then you can break the system while retiring the flag and making the new path permanent.
- blntechie 6y ago
  > I suspect it’s the same quality of thought that can’t fathom that a test could have a bug. I have to admit that I often find it funny when some talk about unit tests being the end of all. No manual QA needed, no staging needed, tests passed and everything is good to roll to production. Unit tests are important, no question but it’s still a piece of code which could have bugs or not enough scenarios covered.
robjan 6y ago
Yep, your feature flag is probably not going to prevent data from getting into an inconsistent state. Maybe it's possible in a system with very loosely related entities but for most non-trivial systems this isn't true.
closeparen 6y ago
It’s a lot easier to test correctness at N commits than at N commits * 2^M feature flag configurations.
oxfordmale 6y ago
The risk of feature flags is well illustrated by the downfall of Knight Capital Group: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ https://dougseven.com/2014/04/17/knightmare-a-devops-caution...
- skissane 6y ago
  > The code that that was updated repurposed an old flag that was used to activate the Power Peg functionality A practice you can adopt which can prevent this: suppose you use JIRA (the practice works the same for any such system, I'm just using JIRA as an example), and for every new feature you must have a JIRA issue, e.g. FEAT-1234. Now you have the rule that the feature flag name must embed both (1) the year and quarter the feature was merged and (2) the JIRA issue. So as part of FEAT-1234 I create feature flag "2020q3-FEAT-1234-split-big-trades". And then you have a custom field in JIRA called "feature flag name", and you record "2020q3-FEAT-1234-split-big-trades" in that field for FEAT-1234. And then you need to have this as a standard check in processes such as code review and release management. It might even be useful to have a report which compares JIRA to the feature flag system to check this rule is being followed. If they'd adopted this practice, that particular bug would likely have been avoided. Adding a new feature, you would not be allowed to reuse an old feature flag from years ago, for the year and quarter and the JIRA ticket would all be different.

by awill 6y ago

aussiedude 6y ago

Agree 100%. Just worked on a incident last week which was caused by feature flags.

NickNameNick 6y ago
The Knight Capital bankruptcy was caused in part by an error related to a feature flag, Which cost them ~US$440 million in trading errors in ~45 minutes.

necovek 6y ago

I would actually venture to say that the way they are usually perused, they increase the risk! Common approach is to just plaster the codebase with "if feature_flag(x)", so you complicate dozens of files/modules/functions, make them harder to reason about and debug. And cleaning it up is commonly a mess. Another approach is to duplicate code which frequently leaves one branch untested. FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master. And even then the complexity they introduce is sometimes a burden, but overall it's worth it over alternative approaches. And sure, you can start implementing stuff iteratively, in small chunks and perusing feature flags to hide bits of unfinished user flow, but the benefits you are getting there are from the former (iterative, small changes), rather than from feature flags. It is still no substitution for testing in a prod-like environment, at the very least that you are not breaking any of the integration places which are not covered by tests (and integration and system tests are slow, which always puts a boundary on how many you can have).

mnsc 6y ago
>FWIW, I do like feature flags, but as a solution for a different problem: long-lived feature branches (forks) of the master But isn't this case what staging originally tried to solve? "this big feature isn't ready for production but we need to regression test in a setup as similar to production as possible" ?
- necovek 6y ago
  Not that I know: if you needed a "staging" for each of the feature branches, the complexity and cost grew, but problems remained (i.e. merging trunk and a branch back and forth). For the last 15 years I've been at teams using a "staging", it was simply a single production-like environment (well, my first project was exactly like production on slightly less beefy hardware with sanitized production DB import nightly) that gets the "trunk" (or "main") deployed to automatically (on my current team, it can be used to run non-production code as part of the deployment process which you bail out of). I am pretty sure there is no one true definition of staging, so it's a spectrum rather than an exact way to do things.
  - mnsc 6y ago
    I'm talking about the prehistoric time before git and feature branches. Where all the developers hacked away for months on That New Feature and all code just went into trunk, cause that's where the development is. Back then staging was invaluable to get the confidence to do that (bi?)annual major release.
    - necovek 6y ago
      Sure, that's the time we learned why feature branches and not releasing often sucks :) And yes, you sometimes had "staging" track your trunk, when you would be missing a staging environment for production where you'd want to test critical fixes, so ultimately, you needed both then.
- loopz 6y ago
  Staging has always been about QA before deployment in production. For tools you buy, one might just end up forgoing staging where you know it must work. Though, what to do if an update bricks the system? For any non-trivial and complex software, you still end up with having at least one test-environment though. Feature flags is about multiple branching. Staging doesn't really give you that in full, just by hacking the environment. It's then that you end up with environments that don't properly reflect future production and start missing stuff. The author sounds like someone familiar with testing, that needs to learn about CI/CD. Sure, for trivial front end stuff, you can often get away with doing it in production. But without synchronization between environments, you end up in trouble and take on more risk, just for the luxury of treating your paying customers as beta-testers. It's a valid opinion depending on what one is working on, but I bet it doesn't apply for most outside unimportant side-projects.

hinkley 6y ago

I suspect it’s the same quality of thought that can’t fathom that a test could have a bug. Implementing a flag takes fewer lines of code than the feature, and so while the odds are very high that the bug will be within a feature, there is a non zero chance that the bug is in the flag. A bug might break a new feature. It might break the old behavior. It may turn the system into a latch (you can launch it darkly, turn it on, and then not be able to turn it off). And then you can break the system while retiring the flag and making the new path permanent.

blntechie 6y ago
> I suspect it’s the same quality of thought that can’t fathom that a test could have a bug. I have to admit that I often find it funny when some talk about unit tests being the end of all. No manual QA needed, no staging needed, tests passed and everything is good to roll to production. Unit tests are important, no question but it’s still a piece of code which could have bugs or not enough scenarios covered.

robjan 6y ago

Yep, your feature flag is probably not going to prevent data from getting into an inconsistent state. Maybe it's possible in a system with very loosely related entities but for most non-trivial systems this isn't true.

closeparen 6y ago

It’s a lot easier to test correctness at N commits than at N commits * 2^M feature flag configurations.

oxfordmale 6y ago

The risk of feature flags is well illustrated by the downfall of Knight Capital Group: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ https://dougseven.com/2014/04/17/knightmare-a-devops-caution...

skissane 6y ago
> The code that that was updated repurposed an old flag that was used to activate the Power Peg functionality A practice you can adopt which can prevent this: suppose you use JIRA (the practice works the same for any such system, I'm just using JIRA as an example), and for every new feature you must have a JIRA issue, e.g. FEAT-1234. Now you have the rule that the feature flag name must embed both (1) the year and quarter the feature was merged and (2) the JIRA issue. So as part of FEAT-1234 I create feature flag "2020q3-FEAT-1234-split-big-trades". And then you have a custom field in JIRA called "feature flag name", and you record "2020q3-FEAT-1234-split-big-trades" in that field for FEAT-1234. And then you need to have this as a standard check in processes such as code review and release management. It might even be useful to have a report which compares JIRA to the feature flag system to check this rule is being followed. If they'd adopted this practice, that particular bug would likely have been avoided. Adding a new feature, you would not be allowed to reuse an old feature flag from years ago, for the year and quarter and the JIRA ticket would all be different.