4 ms·

Deployments Not Releases – Get Good at Delivering Software

44 points by kiyanwang 4y ago

carlmr 4y ago
Feature flags are not a panacea for deployment issues, they can even create many issues. If there is any interaction between features you start getting combinatorial explosion of possible code paths, and you all of a sudden get issues that only affect specific customers due to their set of feature flags. Feature flags have intrinsic complexity. Keeping the code harmonized between customers can cause extra effort when you need to redesign pieces of the system to not need alternative code paths. Both of these costs should be weighed against each other.
- re-thc 4y ago
  Yes, I've seen not-so-big product with 100s of feature flags. It's impossible to test for. Just have to hope for the best. No 1 can keep track of them or know what are any more. To make matters worse it can be dynamically enabled based on cookies / headers per customer.
  - dboreham 4y ago
    They should be removed after the feature is deployed and stable. Some folks have a CI task that checks for stale FF and alerts.
    - re-thc 4y ago
      Should is the headline of every incident report :)
  - dgb23 4y ago
    There are two similar things that can easily be conflated as they have some overlap, or can at least be used interchangeably in some cases. For one we have feature flags, which should be used to disable features for certain environments, possibly AB testing and so on. You use those to reduce risk and friction in an ad-hoc fashion. They are meant to be simple booleans in code and short lived. Then we have feature scope, registration, authorization etc. which can be more wide reaching. They should be part of the data model and treated like everything else that is data driven. When we conflate the two, we get into trouble like you describe.
    - carlmr 4y ago
      >They are meant to be simple booleans in code and short lived. Simple? One "simple" boolean often doubles the number of code paths. 10 simple booleans can easily 1024x the number of code paths. Short lived? There's nothing as permanent as a temporary measure.
tyingq 4y ago
It feels like the article should mention the other side of feature flags. The benefit is obvious, but it does also create other problems. Like, 8 boolean feature flags means 256 true/false combinations that might not all have been tested.
jacquesm 4y ago
Feature flags can work if you are religious about retiring them as soon as possible otherwise you will end up with just another layer of complexity. Ideally by the next release they should be gone unless a feature flag is the gate to an upsell. And then the flag should be on the upsell, not on any particular feature! Featureflags should never be used for privileged operations assigned to particular users.
emilecantin 4y ago
I guess this might work in a pure SaaS environment, but our product has an on-premise version (think Jira data-center), so what keeps us from releasing more often is the maintenance burden that imposes on our customers.
- JohnFen 4y ago
  > what keeps us from releasing more often is the maintenance burden that imposes on our customers. You guys rock. Almost no software companies pay any attention to the fact that every new release imposes a burden on their customers, and rapid release schedules impose that burden rapidly. In a way, it's offloading some of the development burden onto your customers. This is true for pure SaaS products as well, although to a lesser degree. I am very happy to hear of a company that remembers this and takes it into consideration.
- gnat 4y ago
  We release new on-prem versions every fortnight and it's up to the customer to decide if they want to upgrade to it or not. We ack the maintenance burden but also that if they want a new shiny then they need to accept that it means an upgrade. Downside, of course, is that different customers have different versions of our software on their premises. Support's job is harder, as they need to know "oh, you're on 2.14 and this is a problem with preorders, that bug was fixed in 2.18" for a bunch of historical versions and problems.
roel_v 4y ago
Maybe this works in the type of work this guy does, but I've been horribly burned by feature flags. How do you test for the combinatorial explosion of feature interactions? Even if you separate into 'silos' that pinky promise do not affect anything beyond some 'defined' boundary, interactions are still too numerous to test as different products very quickly. And now you have person A using 'feature flag combination' A' and person B using feature flag combination B', and they ask you a question about how this or that is supposed to work, and you won't even know until you are able to exhaustively reason through what the effect of every feature flag this person has en/disabled is - if you even know that at all. Yes in the end it's all in the process, but feature flags are not the end-all be-all these few paragraphs posit them to be.
- capableweb 4y ago
  They are supposed to be removed quickly. The biggest issue I've seen with implementations of feature flags in products are that they are not removed quickly enough. They should be removed as quickly as possible, to avoid what you're saying. In some places, I've even advocated and implemented strategies for not considering something actually done until the feature flag has been removed. So it can be deployed and activated in production, but in terms of project management, the overall task of the feature shouldn't be considered done until cleanup has been done. This process has been the most successful. Another strategy that kind of worked but not as well, is requiring the cleanup of the feature flag to happen in the next sprint. But it tends to be de-prioritized, and sometimes even pushed to the next sprint as other things are always more important. Another strategy that I've discussed and thought about but never actually implemented or advocated for, is limiting the max amount of active feature flags in the specific project to N numbers (ideally 1). So if you want to activate another feature flag, you have to first cleanup the previous one.
  - capn_duck 4y ago
    If they are limited to 1, then they are not appropriate to use as a deployment mechanism. Do you only have one change being worked at a time? On my team, there's typically at least have a dozen different stories being worked in parallel.

44 points by kiyanwang 4y ago

Feature flags are not a panacea for deployment issues, they can even create many issues. If there is any interaction between features you start getting combinatorial explosion of possible code paths, and you all of a sudden get issues that only affect specific customers due to their set of feature flags. Feature flags have intrinsic complexity. Keeping the code harmonized between customers can cause extra effort when you need to redesign pieces of the system to not need alternative code paths. Both of these costs should be weighed against each other.

re-thc 4y ago
Yes, I've seen not-so-big product with 100s of feature flags. It's impossible to test for. Just have to hope for the best. No 1 can keep track of them or know what are any more. To make matters worse it can be dynamically enabled based on cookies / headers per customer.
- dboreham 4y ago
  They should be removed after the feature is deployed and stable. Some folks have a CI task that checks for stale FF and alerts.
  - re-thc 4y ago
    Should is the headline of every incident report :)
- dgb23 4y ago
  There are two similar things that can easily be conflated as they have some overlap, or can at least be used interchangeably in some cases. For one we have feature flags, which should be used to disable features for certain environments, possibly AB testing and so on. You use those to reduce risk and friction in an ad-hoc fashion. They are meant to be simple booleans in code and short lived. Then we have feature scope, registration, authorization etc. which can be more wide reaching. They should be part of the data model and treated like everything else that is data driven. When we conflate the two, we get into trouble like you describe.
  - carlmr 4y ago
    >They are meant to be simple booleans in code and short lived. Simple? One "simple" boolean often doubles the number of code paths. 10 simple booleans can easily 1024x the number of code paths. Short lived? There's nothing as permanent as a temporary measure.

tyingq 4y ago

It feels like the article should mention the other side of feature flags. The benefit is obvious, but it does also create other problems. Like, 8 boolean feature flags means 256 true/false combinations that might not all have been tested.

jacquesm 4y ago

Feature flags can work if you are religious about retiring them as soon as possible otherwise you will end up with just another layer of complexity. Ideally by the next release they should be gone unless a feature flag is the gate to an upsell. And then the flag should be on the upsell, not on any particular feature! Featureflags should never be used for privileged operations assigned to particular users.

emilecantin 4y ago

I guess this might work in a pure SaaS environment, but our product has an on-premise version (think Jira data-center), so what keeps us from releasing more often is the maintenance burden that imposes on our customers.

JohnFen 4y ago
> what keeps us from releasing more often is the maintenance burden that imposes on our customers. You guys rock. Almost no software companies pay any attention to the fact that every new release imposes a burden on their customers, and rapid release schedules impose that burden rapidly. In a way, it's offloading some of the development burden onto your customers. This is true for pure SaaS products as well, although to a lesser degree. I am very happy to hear of a company that remembers this and takes it into consideration.
gnat 4y ago
We release new on-prem versions every fortnight and it's up to the customer to decide if they want to upgrade to it or not. We ack the maintenance burden but also that if they want a new shiny then they need to accept that it means an upgrade. Downside, of course, is that different customers have different versions of our software on their premises. Support's job is harder, as they need to know "oh, you're on 2.14 and this is a problem with preorders, that bug was fixed in 2.18" for a bunch of historical versions and problems.

roel_v 4y ago

Maybe this works in the type of work this guy does, but I've been horribly burned by feature flags. How do you test for the combinatorial explosion of feature interactions? Even if you separate into 'silos' that pinky promise do not affect anything beyond some 'defined' boundary, interactions are still too numerous to test as different products very quickly. And now you have person A using 'feature flag combination' A' and person B using feature flag combination B', and they ask you a question about how this or that is supposed to work, and you won't even know until you are able to exhaustively reason through what the effect of every feature flag this person has en/disabled is - if you even know that at all. Yes in the end it's all in the process, but feature flags are not the end-all be-all these few paragraphs posit them to be.

capableweb 4y ago
They are supposed to be removed quickly. The biggest issue I've seen with implementations of feature flags in products are that they are not removed quickly enough. They should be removed as quickly as possible, to avoid what you're saying. In some places, I've even advocated and implemented strategies for not considering something actually done until the feature flag has been removed. So it can be deployed and activated in production, but in terms of project management, the overall task of the feature shouldn't be considered done until cleanup has been done. This process has been the most successful. Another strategy that kind of worked but not as well, is requiring the cleanup of the feature flag to happen in the next sprint. But it tends to be de-prioritized, and sometimes even pushed to the next sprint as other things are always more important. Another strategy that I've discussed and thought about but never actually implemented or advocated for, is limiting the max amount of active feature flags in the specific project to N numbers (ideally 1). So if you want to activate another feature flag, you have to first cleanup the previous one.
- capn_duck 4y ago
  If they are limited to 1, then they are not appropriate to use as a deployment mechanism. Do you only have one change being worked at a time? On my team, there's typically at least have a dozen different stories being worked in parallel.