4 ms·

I'm surprised to see that Boolean feature flags are common. We almost always ramp up percentages of UUID space or randomly chosen requests, to reduce the blast

I'm surprised to see that Boolean feature flags are common. We almost always ramp up percentages of UUID space or randomly chosen requests, to reduce the blast radius of a bad change.

cdcarter 6y ago
At $JOB, we have feature flags of innumerable shapes and sizes. Some are based on account standing, some are % gradual rollout at random, others are a more thoughtful low-risk to high-risk rollout across customers and hosts. Some are manual flipped per customer/only on certain dev hosts. Literally anything you can think of, we have tied behavior to it. But, we've got good frameworks in place such that at the call sites where behavior diverges, it's just checking a boolean. if PermissionController.get().get(MyPerm.class): doA() else: doB(). I suspect this is pretty common, and its still easy to do the dead code elimination on.
- fennecfoxen 6y ago
  > if PermissionController.get().get(MyPerm.class): doA() else: doB(). oh, but if a behavior changes when a feature flag is active, there's a very strong case for it to be pluggable behavior strategy, so I like these so much better as an unconditional call to `self.getThingStrategy().execute()`
  - anamexis 6y ago
    Won't `getThingStrategy()` do something like `if PermissionController.get().get(MyPerm.class): doA() else: doB()` ?
    - rlayton2 6y ago
      Yes, but you move the logic for deciding which branch to take to an underlying function. It pollutes the parent function less, but results in more overall functions for places to hide. Also, if the logic gets more complicated than just an if statement (if Permision... and date < cutoffdate: etc) you don't further pollute the parent function.
    - fennecfoxen 6y ago
      Naah, but at least that would make the decision simple and clear (if condition return doA else return doB). Better would be for whatever ThingFactory or getThingService instantiates the Thing to make the decision, and compute it up front. If-else statements in application logic tangle the concepts of "what should be done and why?" with "let me do this Way 1" and "let me do this Way 2". Ideally a typical service (or model or similar) shouldn't be aware that "feature flags" as a concept exist, and this should be regarded as inimical to their encapsulation. It should just know that it delegates a decision to Way N.
      - vanviegen 6y ago
        That sounds a lot like enterprisy Java programming. I'm not always thrilled to work with the results of that. :-)
solidasparagus 6y ago
How do you test without a feature flag?
- erik_seaberg 6y ago
  The flag becomes a percentage. The feature isn't always off or always on, it's on for a small fraction of workload, and then that fraction grows as you demonstrate it's safe. Ideally you want metrics that tell you whether the experiment is worse or better than the control group.
  - wahlrus 6y ago
    What you're describing is a common way of using feature flags—except the percentage part comes from how you manage the servers running the binary with config. I.e. on day one, 5% of servers in cluster get True for the flag value. The double the percentage every day until 100% or otherwise rollback if it's a bad cut.
    - erik_seaberg 6y ago
      Partitioning by instance works if you have enough instances to avoid big increases, but at that point you can just deploy known-good and new-feature builds. Runtime checking helps if it's a lot faster than rolling back to the known-good build, or if you're doing concurrent experiments (you may not have enough instances to try every possible combination).
      - RSZC 6y ago
        Having done it both ways this would not be my recommendation unless it's necessary - I think it adds a fair amount of complexity. Some considerations: you'll need some sort of storage mechanism for these flags - is that a centralized configuration service for all your services? Maybe just a table in your database? But database / network calls are expensive to be adding to every single time your code executes the path in question - maybe it makes sense for your service to cache these values locally...but then doesn't that lose part of the purpose of 'fast rollbacks'? Maybe instead of a local cache you spin up a redis instance - but what if this goes down? Will all your instances default to the same value? Etc, etc, etc. I'm not saying this approach is bad, only that it has complexity, and I find I generally can get away without it.
novok 6y ago
Many feature flag systems ramp up doing that from the config server side. So when your client requests feature flags, it gets assigned true/false on that basis.

I'm surprised to see that Boolean feature flags are common. We almost always ramp up percentages of UUID space or randomly chosen requests, to reduce the blast radius of a bad change.

cdcarter 6y ago

At $JOB, we have feature flags of innumerable shapes and sizes. Some are based on account standing, some are % gradual rollout at random, others are a more thoughtful low-risk to high-risk rollout across customers and hosts. Some are manual flipped per customer/only on certain dev hosts. Literally anything you can think of, we have tied behavior to it. But, we've got good frameworks in place such that at the call sites where behavior diverges, it's just checking a boolean. if PermissionController.get().get(MyPerm.class): doA() else: doB(). I suspect this is pretty common, and its still easy to do the dead code elimination on.

fennecfoxen 6y ago
> if PermissionController.get().get(MyPerm.class): doA() else: doB(). oh, but if a behavior changes when a feature flag is active, there's a very strong case for it to be pluggable behavior strategy, so I like these so much better as an unconditional call to `self.getThingStrategy().execute()`
- anamexis 6y ago
  Won't `getThingStrategy()` do something like `if PermissionController.get().get(MyPerm.class): doA() else: doB()` ?
  - rlayton2 6y ago
    Yes, but you move the logic for deciding which branch to take to an underlying function. It pollutes the parent function less, but results in more overall functions for places to hide. Also, if the logic gets more complicated than just an if statement (if Permision... and date < cutoffdate: etc) you don't further pollute the parent function.
  - fennecfoxen 6y ago
    Naah, but at least that would make the decision simple and clear (if condition return doA else return doB). Better would be for whatever ThingFactory or getThingService instantiates the Thing to make the decision, and compute it up front. If-else statements in application logic tangle the concepts of "what should be done and why?" with "let me do this Way 1" and "let me do this Way 2". Ideally a typical service (or model or similar) shouldn't be aware that "feature flags" as a concept exist, and this should be regarded as inimical to their encapsulation. It should just know that it delegates a decision to Way N.
    - vanviegen 6y ago
      That sounds a lot like enterprisy Java programming. I'm not always thrilled to work with the results of that. :-)

solidasparagus 6y ago

How do you test without a feature flag?

erik_seaberg 6y ago
The flag becomes a percentage. The feature isn't always off or always on, it's on for a small fraction of workload, and then that fraction grows as you demonstrate it's safe. Ideally you want metrics that tell you whether the experiment is worse or better than the control group.
- wahlrus 6y ago
  What you're describing is a common way of using feature flags—except the percentage part comes from how you manage the servers running the binary with config. I.e. on day one, 5% of servers in cluster get True for the flag value. The double the percentage every day until 100% or otherwise rollback if it's a bad cut.
  - erik_seaberg 6y ago
    Partitioning by instance works if you have enough instances to avoid big increases, but at that point you can just deploy known-good and new-feature builds. Runtime checking helps if it's a lot faster than rolling back to the known-good build, or if you're doing concurrent experiments (you may not have enough instances to try every possible combination).
    - RSZC 6y ago
      Having done it both ways this would not be my recommendation unless it's necessary - I think it adds a fair amount of complexity. Some considerations: you'll need some sort of storage mechanism for these flags - is that a centralized configuration service for all your services? Maybe just a table in your database? But database / network calls are expensive to be adding to every single time your code executes the path in question - maybe it makes sense for your service to cache these values locally...but then doesn't that lose part of the purpose of 'fast rollbacks'? Maybe instead of a local cache you spin up a redis instance - but what if this goes down? Will all your instances default to the same value? Etc, etc, etc. I'm not saying this approach is bad, only that it has complexity, and I find I generally can get away without it.

novok 6y ago

Many feature flag systems ramp up doing that from the config server side. So when your client requests feature flags, it gets assigned true/false on that basis.