55 ms·

This exactly. Feature flags are necessary for rollouts, but they also quickly become technical debt. Multiple outages happened at my last company when the featu

by yojo 5y ago

This exactly. Feature flags are necessary for rollouts, but they also quickly become technical debt. Multiple outages happened at my last company when the feature flag service went down and old flags defaulted to “off”. It got to the point that the supporting team restricted flag-based rollouts to 95%. If you wanted it on for everyone, you had to remove the flag. Not sure if I fully agree with that design, but as an organization we were consistently failing to clean up flags.

ford 5y ago
It sounds like a poor design choice to not give clients the ability to set their own defaults
- Minor49er 5y ago
  This really depends on the use case. For example, consider a platform where users can sign up to post or listen to music. There might be three types of user: "listener", "artist", and "label". There may be permissions and configuration settings that would cater to each one of these types within the same interface, and maybe in some exceptional cases, they could be intermixed (eg: while an artist and label can post music, maybe an artist can also track stats about what they like to listen to, something that only a listener account can do normally). The platform might want to configure each of these to streamline usage and to remove features that aren't relevant to that account type, but some accounts may be specialized for whatever reason.
- yojo 5y ago
  Config level defaults were set in the flag service. But when that goes down, the client has to decide what to do. I think there was an optional arg for specifying default behavior, but even when used there’s the problem that the default you pick for a rollout will be “off”. If you don’t go back and change it later, you’re back in the same boat.
nicoburns 5y ago
Sounds like you have too many services if you have one just for feature flags! At my company we bundle feature flagging into the main auth service, if you can login then you can access your feature flags.
- yojo 5y ago
  The feature flag service was doing double duty as both the “marketing experimentation A/B testing” platform and the “feature rollout” platform. Not defending the service boundaries, but it had enough meat by itself that there was a full time team working on it. I also think the intent of the service was to be used for temporary experiments and rollouts, with the assumption that everything in the system could safely turn off. If these assumptions held you might not want to allow a bug in the flag service to take down auth. Indeed, the outages we did have from feature flag service going down would have been worse outages if it was crashing all of auth and unaffected clients couldn’t log in. In practice long running tech debt resulted in unexercised code paths that might be years out of date.
  - nicoburns 5y ago
    > The feature flag service was doing double duty as both the “marketing experimentation A/B testing” platform and the “feature rollout” platform. Not defending the service boundaries, but it had enough meat by itself that there was a full time team working on it. Ah, we're small enough that we don't have a marketing team and we don't have capacity to do A/B testing either!
    - yojo 5y ago
      Ah yeah. The right answer to this question looks very different depending on how old/mature your codebase is and how big your eng team is. My example is from a ~1200 person eng org with a ten year old codebase.

by yojo 5y ago

ford 5y ago

It sounds like a poor design choice to not give clients the ability to set their own defaults

Minor49er 5y ago
This really depends on the use case. For example, consider a platform where users can sign up to post or listen to music. There might be three types of user: "listener", "artist", and "label". There may be permissions and configuration settings that would cater to each one of these types within the same interface, and maybe in some exceptional cases, they could be intermixed (eg: while an artist and label can post music, maybe an artist can also track stats about what they like to listen to, something that only a listener account can do normally). The platform might want to configure each of these to streamline usage and to remove features that aren't relevant to that account type, but some accounts may be specialized for whatever reason.
yojo 5y ago
Config level defaults were set in the flag service. But when that goes down, the client has to decide what to do. I think there was an optional arg for specifying default behavior, but even when used there’s the problem that the default you pick for a rollout will be “off”. If you don’t go back and change it later, you’re back in the same boat.

nicoburns 5y ago

Sounds like you have too many services if you have one just for feature flags! At my company we bundle feature flagging into the main auth service, if you can login then you can access your feature flags.

yojo 5y ago
The feature flag service was doing double duty as both the “marketing experimentation A/B testing” platform and the “feature rollout” platform. Not defending the service boundaries, but it had enough meat by itself that there was a full time team working on it. I also think the intent of the service was to be used for temporary experiments and rollouts, with the assumption that everything in the system could safely turn off. If these assumptions held you might not want to allow a bug in the flag service to take down auth. Indeed, the outages we did have from feature flag service going down would have been worse outages if it was crashing all of auth and unaffected clients couldn’t log in. In practice long running tech debt resulted in unexercised code paths that might be years out of date.
- nicoburns 5y ago
  > The feature flag service was doing double duty as both the “marketing experimentation A/B testing” platform and the “feature rollout” platform. Not defending the service boundaries, but it had enough meat by itself that there was a full time team working on it. Ah, we're small enough that we don't have a marketing team and we don't have capacity to do A/B testing either!
  - yojo 5y ago
    Ah yeah. The right answer to this question looks very different depending on how old/mature your codebase is and how big your eng team is. My example is from a ~1200 person eng org with a ten year old codebase.