6 ms·

Feature Toggles

66 points by okfine 11y ago

cpitman 11y ago
Overall feature toggles are an interesting idea. I'm not quite as adverse to long lived branches as some seem to be, especially with CI systems that can automatically detect merge conflicts when they first occur and run automated tests on the merged code base. However, this part seems strange to me: > The team particularly appreciate that this will allow them to test their new algorithm without needing a separate testing environment. Even with feature toggles, I would never want to test a new feature for the first time in my production environment. Sure, the feature isn't on for everyone. But it still can have a performance defect that brings the overall system to its knees or a functionality defect that corrupts the data store for everyone.
- benjiweber 11y ago
  I find it amusing when CI tooling is used like this, to allow people to keep changes isolated in separate branches with less pain from delaying the integration. It's more Continuous Isolation than Continuous Integration. There are ways to test features for performance/accuracy in the production environment with reduced risk. One approach I've used successfully is branch by abstraction with verification. http://www.alwaysagileconsulting.com/articles/application-pattern-verify-branch-by-abstraction/ http://www.alwaysagileconsulting.com/articles/application-pa... Essentially you first extract a common interface for the component you are replacing. Then you release both versions and send (all or a percentage of) input events to both the old and the new implementation. You discard the response from the new implementation and continue using the old codepath for responding to users / performing calculations, but importantly - you compare the old and new results and alert/log if they differ. This allows you to gain confidence in a new implementation's behaviour in the production environment and integrate your code with the rest of the system with significantly reduced risk. When you're happy you can flip over to the new implementation and delete the old. There are also various other techniques for reducing the risk of datastore corruption that you mention. I've written about some of them here http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-of-data-damage/ http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-... Most of these risks are actually smaller with more frequent and smaller releases of functionality into production. Releasing more regularly also encourages you to think about how to properly mitigate these risks.
- sambe 11y ago
  There is a substantial section of the article about modifying their high level tests to test both branches and keeping the toggle off in production whilst turning it on for some exploratory testing.
- qwer 11y ago
  > I'm not quite as adverse to long lived branches as some seem to be, especially with CI systems that can automatically detect merge conflicts when they first occur and run automated tests on the merged code base. This system of branching is clearly the best way to maximize large merge conflicts, if you think about it. Let me restate this: The main reason that you branch (to avoid conflicts with other devs) is the single scenario that this system does the absolute worst at. Conversely, if you're not having problems with this system, you never needed it in the first place.
egometry 11y ago
At IMVU we had feature toggles rolled into the A/B Experiment system. It proved to be a vital part of Continuous Deployment. Basically: Wrap every new feature in an Experiment Toggle. Deploy to production (once ready for 'non sandbox' testing). Turn on for test users, test. If good, start the rollout to experimental users. If after weeks the numbers worked out, turn it on for everyone... ...but if at any point unexpectedly bad things happened, just flip the feature toggle switch. Most of the time if "bad things" were happening in a relatively new feature, this would stop the panic and let analysis of the root cause of the problem happen without the pressure of everything being on fire.
- prostoalex 11y ago
  Same at Facebook https://www.quora.com/How-does-Facebooks-Gatekeeper-service-work/answer/Jack-Lindamood https://www.quora.com/How-does-Facebooks-Gatekeeper-service-... with rollout UI similar to what advertisers see on ad purchasing UI (by country, college, demographics, etc.)
jbardnz 11y ago
I really like the idea of feature toggles but i'm scared of quickly having a lot of technical debt because of them. I was looking at using something like https://launchdarkly.com https://launchdarkly.com but the pricing is very expensive even at fairly small scale.
- jkodumal 11y ago
  Co-founder of LaunchDarkly here-- we're dropping our startup package to $9 / month in the next week. Come check us out.
- w-ll 11y ago
  If you have a RBAC setup it's pretty easy to setup feature testing. It's the second most awesome reason why I setup RBACs on any and all projects.
- junto 11y ago
  We use feature toggled a fair amount on our SAAS application. It does create technical debt which at some point has to be addressed. One thing that confuses our junior developers is the difference between a feature toggles and a client permission. Often they misuse a feature toggle as a client permission, simply because only one client is using that feature (currently).
  - justinucd 11y ago
    How do you typically address or mitigate your technical debt?
    - junto 11y ago
      Currently we are forced to ignore it until it becomes a performance problem or a new feature touches the same code. This is simply because new features are paid for by clients and they take priority over technical debt. We only tend to refactor code (as a rule) when you are already working inside that code section on something new (feature, improvement or bug). Otherwise junior developers have a tendency to want to refactor all the things! If you wrote a new unit test, or altered an existing one to accommodate new code, then knock yourself out and refactor away and knock yourself out!
quanticle 11y ago
An interesting way to implement feature toggles is with your existing A/B test system. Basically, create an "experiment" for each new feature. Set the initial "treatment" size to something like 10%. Once the feature has been validated in production, set the treatment size to 100%.
- tamana 11y ago
  It's scary that you have to explain this. This is A/B testing 102, all new features are experiments.
omnibrain 11y ago
It's interesting to see this here now, because ~10 hours ago, when it came up in my feedly stream, I tried to submit it and only got the message that it's already submitted. I searched here for it but could not find it, though.
- omnibrain 11y ago
  Thanks for the downvotes for being curious how Hacker News works. :(
rbranson 11y ago
A few months ago there was a great blog post by the Instagram team on their particularly advanced flavor of this for their continuous integration. Highly recommend: https://engineering.instagram.com/posts/496049610561948/flexible-feature-control-at-instagram/ https://engineering.instagram.com/posts/496049610561948/flex...
- jessaustin 11y ago
  Whoa. That's an ugly URL! b^) I clicked through just to see if I could cut out the random number, but no dice.
  - pkaeding 11y ago
    It seems the 'random' number is actually the important bit. You can remove the `/flexible-feature-control-at-instagram/` part, which is only there for SEO (presumably). StackOverflow URLs work the same way, in fact...
isbadawi 11y ago
There was a nice talk about feature toggles at RubyConf 2015: https://www.youtube.com/watch?v=rBBLMmr9e-k https://www.youtube.com/watch?v=rBBLMmr9e-k
jammycakes 11y ago
One point to be aware of with feature toggles: if you're using them as a substitute for branching and merging in your VCS, you're deploying code into production that you know for a fact to be buggy, insufficiently tested, and incorrect. If your feature toggles themselves have bugs, or don't properly isolate your new code, you could easily run into problems up to and including data corruption. You may be switching your controllers on and off, but are you also toggling your static assets such as HTML, CSS and JavaScript? People are often scared of feature branches because of large, tricky merge conflicts. However, large, tricky merge conflicts warn you (noisily) of problems such as these before your code gets into production. Problems with feature toggles, on the other hand, don't show up until after your code gets into production, by which time it may well be too late.
xorcist 11y ago
Architecture people tend to speak in trivial examples. Swapping out an algorithm with a backwards compatible one? Good for you! It doesn't matter how you do it because it's the most trivial example one could think of. (A more realistic example would be new functionality that touches large parts of the code base, or compliance with a new API that the payment processor switches that has more data on every payment as it crosses our systems.) I never understood why some people are scared of branching. It seems to work pretty well for Linux development. Is your project really that much more complex? As long as you merge continously from upstream, you're golden. What's the problem you're trying to solve, exactly? I _do_ have some reservations against #ifdef-sprinkled code. And it's not an unlikely scenario that your feature toggles turn into something very similar, if conditions change and maybe they're not that temporary anymore.
- samwgoldman 11y ago
  I think the calculus changes when you consider teams with 1000s of programmers committing 100s of changes per day. Keeping feature branches up-to-date can pretty quickly adds up to a lot of time and creates a lot of friction for refactoring/cleanup-type changes. Feature toggles enable some additional cool stuff like partial/incremental rollout and A/B testing, which really pay dividends.
  - lucaspiller 11y ago
    I've never worked on codebases with so many programmers, but I kind of feel that's if that's how you work, 'you're doing it wrong'. Why would you have so many programmers working on a single codebase, why not split it up in distinct parts? If that's not possible because so much stuff is shared, how can you possibly work effectively with so many people changing so many things?
    - geocar 11y ago
      1000 programmers on a single codebase means: • "master" always builds • "master" is always stable Any testing has to be done with in-code branches, and has to be built in-place incrementally so that other developers can see you touching stuff (and can be defensive about their tasks).
tomcam 11y ago
Ah, isn't this just... an option?
yosidahan 11y ago
Configo.io can do that for your mobile app!
yosidahan 11y ago
Configo.io can do that for your mobile app
gioele 11y ago
I find the `scientist` gem by GitHub [1] a very nice way to approach the problem of testing alternative implementations. Basically `scientist` always runs both the original code and the new code. def allows?(user) science "widget-permissions" do |e| e.use { model.check_user(user).valid? } # old way e.try { user.can?(:read, model) } # new way end # returns the control value end It returns the result given by the original code, but it also compares both results and store statistics about when the two codes produce different results. This allows you to run experiments directly in the deployed branch without any service disruption. [1] https://github.com/github/scientist https://github.com/github/scientist
eps 11y ago
Oi vey. Coming from the good old C I am frankly baffled how some triviality like this one can be wrapped into so many fancy words and a full article. This is like hearing that taking your shoes off when entering home is a good idea. Do you know why? Let's start with an example. Picture this - it's raining outside, water soaks the earth and some of it may get on the soles of your shoes... you get the idea. It's a subject hardly worth a passing mention if any. "Feature toggles". Bah. Makes you wonder what will happen when they discover function pointers or, god forbid, the config files. Now that will likely be worthy of at least a book.
geocar 11y ago
A more powerful version of these things are called dynamic variables, and they're worth some study beyond feature selection and testing: Otherwise you just get global variables with funny names. Many languages have some dynamic variables (this, self), and few languages have first-class support for them (notably CL, Perl). While dynamic variables are tricky to emulate fully, as long as you have the ability to catch exceptions and closures, a very fair simulation can be made: (function(){ var o={}; D=function(k){return o[k]} dlet=function(b,f){ var r,s={};function oops(){for(var k in s)o[k]=s[k]}; for(var k in b)s[k]=o[k],o[k]=b[k]; try{r=f()}catch(e){oops();throw e;}; oops();return r; }; })(); Besides feature variables that are globally accessible, one useful thing you can do with dynamic variables is set up error handlers. Consider the situation where you are saving a big file to the disk and run out of space. If you: save_stuff(); if(oops)throw 'out of space'; save_more_stuff(); then your slow saving operation needs to be retried from the start, however if you use a dynamic variable to look up the handler, you can: save_stuff(); if(oops)D('out of space')(next); else next(); function next(){ save_more_stuff(); } The user can then use their fancy multitasking environment to clean up some space, and we can continue our operation. I generally recommend this form of error handling anyway. Another useful thing is for context: Imagine you have a user interface choice between an HTTP response and a command line interface. One way to do this is to have two applications, and another way is DI (dependancy injection: pass the "response" object around), however another way is using dynamic variables: D("output")(...); This has the benefit of not requiring an extra argument to all of your functions. This happens more than you might think: Many people when confronted with all their dynamic variables put them into some kind of "master object" (current logged in user, output handler, database settings, etc), however dynamic variables When you do this, a lot of features become very easy to set: • Capturing output (simply mock a new "output" var) • Testing logic: dlet({db:test_settings},r)===dlet({db:live_settings},r) • Impersonate users (if the right credentials are available) And so on.
bauc 11y ago
Feature toggles are useful but should be used with great care, especially in complex codebases. There are challenges to using them. If you aren't careful or don't have very good test coverage you can break unknown sections of code. The other pain is removing them, again if not done carefully or with very good test coverage you can unknowingly break things.
evvvvr 11y ago
This all thing reminds me why I love so much so called Enterprise Developers and Martin Fowler with Thoughtworks and their articles (no offense, thanks to Martin for the idea of refactoring, but such things are wrong). Seriously, they take simple idea reducible to more general principles which is not a very big deal and start build ad-hoc ideology and terminology around it: Hm-m, some abstraction around feature flags config? I think we should call it "toggle router". Then people start to create frameworks (not libraries) for this and after all technical recruiters will end-up searching for people with particular "toggle router experience".
celphys 11y ago
I am pretty disapointed by the limited use cases pictured in the article. I pinged the author without success. Check at bottom of ff4.org page 10 others. E.g : Deliver your product with toggles off on each node of your cluster and when all are ready : toggle on - no more inconsistency during release - no rollback as you simply toggle off if not working.....

66 points by okfine 11y ago

Overall feature toggles are an interesting idea. I'm not quite as adverse to long lived branches as some seem to be, especially with CI systems that can automatically detect merge conflicts when they first occur and run automated tests on the merged code base. However, this part seems strange to me: > The team particularly appreciate that this will allow them to test their new algorithm without needing a separate testing environment. Even with feature toggles, I would never want to test a new feature for the first time in my production environment. Sure, the feature isn't on for everyone. But it still can have a performance defect that brings the overall system to its knees or a functionality defect that corrupts the data store for everyone.

benjiweber 11y ago
I find it amusing when CI tooling is used like this, to allow people to keep changes isolated in separate branches with less pain from delaying the integration. It's more Continuous Isolation than Continuous Integration. There are ways to test features for performance/accuracy in the production environment with reduced risk. One approach I've used successfully is branch by abstraction with verification. http://www.alwaysagileconsulting.com/articles/application-pattern-verify-branch-by-abstraction/ http://www.alwaysagileconsulting.com/articles/application-pa... Essentially you first extract a common interface for the component you are replacing. Then you release both versions and send (all or a percentage of) input events to both the old and the new implementation. You discard the response from the new implementation and continue using the old codepath for responding to users / performing calculations, but importantly - you compare the old and new results and alert/log if they differ. This allows you to gain confidence in a new implementation's behaviour in the production environment and integrate your code with the rest of the system with significantly reduced risk. When you're happy you can flip over to the new implementation and delete the old. There are also various other techniques for reducing the risk of datastore corruption that you mention. I've written about some of them here http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-of-data-damage/ http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-... Most of these risks are actually smaller with more frequent and smaller releases of functionality into production. Releasing more regularly also encourages you to think about how to properly mitigate these risks.
sambe 11y ago
There is a substantial section of the article about modifying their high level tests to test both branches and keeping the toggle off in production whilst turning it on for some exploratory testing.
qwer 11y ago
> I'm not quite as adverse to long lived branches as some seem to be, especially with CI systems that can automatically detect merge conflicts when they first occur and run automated tests on the merged code base. This system of branching is clearly the best way to maximize large merge conflicts, if you think about it. Let me restate this: The main reason that you branch (to avoid conflicts with other devs) is the single scenario that this system does the absolute worst at. Conversely, if you're not having problems with this system, you never needed it in the first place.

egometry 11y ago

At IMVU we had feature toggles rolled into the A/B Experiment system. It proved to be a vital part of Continuous Deployment. Basically: Wrap every new feature in an Experiment Toggle. Deploy to production (once ready for 'non sandbox' testing). Turn on for test users, test. If good, start the rollout to experimental users. If after weeks the numbers worked out, turn it on for everyone... ...but if at any point unexpectedly bad things happened, just flip the feature toggle switch. Most of the time if "bad things" were happening in a relatively new feature, this would stop the panic and let analysis of the root cause of the problem happen without the pressure of everything being on fire.

prostoalex 11y ago
Same at Facebook https://www.quora.com/How-does-Facebooks-Gatekeeper-service-work/answer/Jack-Lindamood https://www.quora.com/How-does-Facebooks-Gatekeeper-service-... with rollout UI similar to what advertisers see on ad purchasing UI (by country, college, demographics, etc.)

jbardnz 11y ago

I really like the idea of feature toggles but i'm scared of quickly having a lot of technical debt because of them. I was looking at using something like https://launchdarkly.com https://launchdarkly.com but the pricing is very expensive even at fairly small scale.

jkodumal 11y ago
Co-founder of LaunchDarkly here-- we're dropping our startup package to $9 / month in the next week. Come check us out.
w-ll 11y ago
If you have a RBAC setup it's pretty easy to setup feature testing. It's the second most awesome reason why I setup RBACs on any and all projects.
junto 11y ago
We use feature toggled a fair amount on our SAAS application. It does create technical debt which at some point has to be addressed. One thing that confuses our junior developers is the difference between a feature toggles and a client permission. Often they misuse a feature toggle as a client permission, simply because only one client is using that feature (currently).
- justinucd 11y ago
  How do you typically address or mitigate your technical debt?
  - junto 11y ago
    Currently we are forced to ignore it until it becomes a performance problem or a new feature touches the same code. This is simply because new features are paid for by clients and they take priority over technical debt. We only tend to refactor code (as a rule) when you are already working inside that code section on something new (feature, improvement or bug). Otherwise junior developers have a tendency to want to refactor all the things! If you wrote a new unit test, or altered an existing one to accommodate new code, then knock yourself out and refactor away and knock yourself out!

quanticle 11y ago

An interesting way to implement feature toggles is with your existing A/B test system. Basically, create an "experiment" for each new feature. Set the initial "treatment" size to something like 10%. Once the feature has been validated in production, set the treatment size to 100%.

tamana 11y ago
It's scary that you have to explain this. This is A/B testing 102, all new features are experiments.

omnibrain 11y ago

It's interesting to see this here now, because ~10 hours ago, when it came up in my feedly stream, I tried to submit it and only got the message that it's already submitted. I searched here for it but could not find it, though.

omnibrain 11y ago
Thanks for the downvotes for being curious how Hacker News works. :(

rbranson 11y ago

A few months ago there was a great blog post by the Instagram team on their particularly advanced flavor of this for their continuous integration. Highly recommend: https://engineering.instagram.com/posts/496049610561948/flexible-feature-control-at-instagram/ https://engineering.instagram.com/posts/496049610561948/flex...

jessaustin 11y ago
Whoa. That's an ugly URL! b^) I clicked through just to see if I could cut out the random number, but no dice.
- pkaeding 11y ago
  It seems the 'random' number is actually the important bit. You can remove the `/flexible-feature-control-at-instagram/` part, which is only there for SEO (presumably). StackOverflow URLs work the same way, in fact...

isbadawi 11y ago

There was a nice talk about feature toggles at RubyConf 2015: https://www.youtube.com/watch?v=rBBLMmr9e-k https://www.youtube.com/watch?v=rBBLMmr9e-k

jammycakes 11y ago

One point to be aware of with feature toggles: if you're using them as a substitute for branching and merging in your VCS, you're deploying code into production that you know for a fact to be buggy, insufficiently tested, and incorrect. If your feature toggles themselves have bugs, or don't properly isolate your new code, you could easily run into problems up to and including data corruption. You may be switching your controllers on and off, but are you also toggling your static assets such as HTML, CSS and JavaScript? People are often scared of feature branches because of large, tricky merge conflicts. However, large, tricky merge conflicts warn you (noisily) of problems such as these before your code gets into production. Problems with feature toggles, on the other hand, don't show up until after your code gets into production, by which time it may well be too late.

xorcist 11y ago

Architecture people tend to speak in trivial examples. Swapping out an algorithm with a backwards compatible one? Good for you! It doesn't matter how you do it because it's the most trivial example one could think of. (A more realistic example would be new functionality that touches large parts of the code base, or compliance with a new API that the payment processor switches that has more data on every payment as it crosses our systems.) I never understood why some people are scared of branching. It seems to work pretty well for Linux development. Is your project really that much more complex? As long as you merge continously from upstream, you're golden. What's the problem you're trying to solve, exactly? I _do_ have some reservations against #ifdef-sprinkled code. And it's not an unlikely scenario that your feature toggles turn into something very similar, if conditions change and maybe they're not that temporary anymore.

samwgoldman 11y ago
I think the calculus changes when you consider teams with 1000s of programmers committing 100s of changes per day. Keeping feature branches up-to-date can pretty quickly adds up to a lot of time and creates a lot of friction for refactoring/cleanup-type changes. Feature toggles enable some additional cool stuff like partial/incremental rollout and A/B testing, which really pay dividends.
- lucaspiller 11y ago
  I've never worked on codebases with so many programmers, but I kind of feel that's if that's how you work, 'you're doing it wrong'. Why would you have so many programmers working on a single codebase, why not split it up in distinct parts? If that's not possible because so much stuff is shared, how can you possibly work effectively with so many people changing so many things?
  - geocar 11y ago
    1000 programmers on a single codebase means: • "master" always builds • "master" is always stable Any testing has to be done with in-code branches, and has to be built in-place incrementally so that other developers can see you touching stuff (and can be defensive about their tasks).

tomcam 11y ago

Ah, isn't this just... an option?

yosidahan 11y ago

Configo.io can do that for your mobile app!

yosidahan 11y ago

Configo.io can do that for your mobile app

gioele 11y ago

I find the `scientist` gem by GitHub [1] a very nice way to approach the problem of testing alternative implementations. Basically `scientist` always runs both the original code and the new code. def allows?(user) science "widget-permissions" do |e| e.use { model.check_user(user).valid? } # old way e.try { user.can?(:read, model) } # new way end # returns the control value end It returns the result given by the original code, but it also compares both results and store statistics about when the two codes produce different results. This allows you to run experiments directly in the deployed branch without any service disruption. [1] https://github.com/github/scientist https://github.com/github/scientist

eps 11y ago

Oi vey. Coming from the good old C I am frankly baffled how some triviality like this one can be wrapped into so many fancy words and a full article. This is like hearing that taking your shoes off when entering home is a good idea. Do you know why? Let's start with an example. Picture this - it's raining outside, water soaks the earth and some of it may get on the soles of your shoes... you get the idea. It's a subject hardly worth a passing mention if any. "Feature toggles". Bah. Makes you wonder what will happen when they discover function pointers or, god forbid, the config files. Now that will likely be worthy of at least a book.

geocar 11y ago

A more powerful version of these things are called dynamic variables, and they're worth some study beyond feature selection and testing: Otherwise you just get global variables with funny names. Many languages have some dynamic variables (this, self), and few languages have first-class support for them (notably CL, Perl). While dynamic variables are tricky to emulate fully, as long as you have the ability to catch exceptions and closures, a very fair simulation can be made: (function(){ var o={}; D=function(k){return o[k]} dlet=function(b,f){ var r,s={};function oops(){for(var k in s)o[k]=s[k]}; for(var k in b)s[k]=o[k],o[k]=b[k]; try{r=f()}catch(e){oops();throw e;}; oops();return r; }; })(); Besides feature variables that are globally accessible, one useful thing you can do with dynamic variables is set up error handlers. Consider the situation where you are saving a big file to the disk and run out of space. If you: save_stuff(); if(oops)throw 'out of space'; save_more_stuff(); then your slow saving operation needs to be retried from the start, however if you use a dynamic variable to look up the handler, you can: save_stuff(); if(oops)D('out of space')(next); else next(); function next(){ save_more_stuff(); } The user can then use their fancy multitasking environment to clean up some space, and we can continue our operation. I generally recommend this form of error handling anyway. Another useful thing is for context: Imagine you have a user interface choice between an HTTP response and a command line interface. One way to do this is to have two applications, and another way is DI (dependancy injection: pass the "response" object around), however another way is using dynamic variables: D("output")(...); This has the benefit of not requiring an extra argument to all of your functions. This happens more than you might think: Many people when confronted with all their dynamic variables put them into some kind of "master object" (current logged in user, output handler, database settings, etc), however dynamic variables When you do this, a lot of features become very easy to set: • Capturing output (simply mock a new "output" var) • Testing logic: dlet({db:test_settings},r)===dlet({db:live_settings},r) • Impersonate users (if the right credentials are available) And so on.

bauc 11y ago

Feature toggles are useful but should be used with great care, especially in complex codebases. There are challenges to using them. If you aren't careful or don't have very good test coverage you can break unknown sections of code. The other pain is removing them, again if not done carefully or with very good test coverage you can unknowingly break things.

evvvvr 11y ago

This all thing reminds me why I love so much so called Enterprise Developers and Martin Fowler with Thoughtworks and their articles (no offense, thanks to Martin for the idea of refactoring, but such things are wrong). Seriously, they take simple idea reducible to more general principles which is not a very big deal and start build ad-hoc ideology and terminology around it: Hm-m, some abstraction around feature flags config? I think we should call it "toggle router". Then people start to create frameworks (not libraries) for this and after all technical recruiters will end-up searching for people with particular "toggle router experience".

celphys 11y ago

I am pretty disapointed by the limited use cases pictured in the article. I pinged the author without success. Check at bottom of ff4.org page 10 others. E.g : Deliver your product with toggles off on each node of your cluster and when all are ready : toggle on - no more inconsistency during release - no rollback as you simply toggle off if not working.....