9 ms·

I've got a much nicer branching model- try not to have one. Everyone works off master, and you aren't allowed to check in code that won't run in production. Hid

by mabbo 9y ago

I've got a much nicer branching model- try not to have one. Everyone works off master, and you aren't allowed to check in code that won't run in production. Hide unfinished features behind feature flags, and never merge/push a change that won't pass tests/CI. The chaos of huge feature merges (a key source of bugs I've experienced) is minimized. You deploy fixes hourly, not weekly (or later monthly when it just won't seem to pass CI). The time between code being written and a bug being seen can be reduced to minutes and hours, making finding the root cause a breeze. Just my preference, but very open to debate.

sillysaurus3 9y ago
Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there. Some concrete questions: - How do you prevent your codebase from becoming if-statement spaghetti? - How do you prevent new features from being leaked to the user? They'll see the new features in the front end source code. At many companies this isn't an acceptable tradeoff. So do you preprocess the release code and strip out the disabled feature flags? With what? - What do you use to control feature flags? Just a json file filled with `"foo-bar-feature": true/false`, or something more sophisticated like a control panel that you can use to say "10% of our users will see this feature"?
- NicoJuicy 9y ago
  Who says no one is embracing them? There are tons of tools available if you check for it. But you should google for "feature toggle" libraries and not feature flags
- mabbo 9y ago
  > How do you prevent your codebase from becoming if-statement spaghetti? A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch, and launch fully or remove the feature; consider 'hiding' the ifs behind factories that build the objects that implement the different logic; also, if you have 20 features is development for the same area of your codebase, worry- you may be trying too much! > How do you prevent new features from being leaked to the user? I haven't had to worry about this very often just based on my projects, but there are strategies. You could be sending the user a different version of the js/html based on the feature flags (preprocessor as you said). Haven't had to do that, but woah that would be a fun little project. > What do you use to control feature flags? My favorite implementation was an S3 json file that effectively encoded a decision tree based on variables used. In code, we could say "here are the five variables about this request, feature object are you enabled?". By modifying that json file, you could change the features state at run time. (Note: this was not perfectly implemented/designed by me and caused a few issues when we first used it. Oops. But you can do very quick solutions too, read a file or make an object that decides based on non-dynamic logic.
  - jasode 9y ago
    >A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch I think a big factor that's unspoken about "trunk-based-master" development is that it fits better with "website apps" such as Facebook and Etsy. The "live website" can be thought of as a "single executable" and trunk-based mental model maps well to that. The feature-flags as a replacement for branches doesn't result in an unusable combinatorial explosion. There are minimal # of "alternate universes" of Facebook... maybe a "Facebook for internal engineers" etc. You can tame that with feature-flags. However, for corporate development where the output is "exe" files... e.g. version 2.x of product has dependencies on a old version of a library and a version 3.x uses a totally different architecture with different conflicting dependencies... you can't model that code evolution in a clean manner with feature-flag "if" statements. If you move the "if" statements outside of the code and into preprocessors (#ifdef) or the "build" system such as cmake, you've just shifted the problem around instead of solving it. Branches instead of feature-flags are cleaner and more sane for certain domains.
    - mabbo 9y ago
      Fortunately, exe development is becoming a small subset of software these days.
- hoorayimhelping 9y ago
  >Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them I worked at Etsy from early 2012 to late 2015 and I used feature flags every day I was there. >How do you prevent your codebase from becoming if-statement spaghetti? You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go. >How do you prevent new features from being leaked to the user? You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns. >What do you use to control feature flags? You can use a service like LaunchDarkly or Optimizely or you can write your own service, or you can have a file in memory. Usually it starts simple with boolean toggles and evolves into a ramp up system that let's you selectively target users. The important part is that you need to be able to change features without redeploying code.
  - sillysaurus3 9y ago
    You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go. I don't understand. I was asking about the development source code, not the code delivered to the user. The dev source code isn't stripped of if-statements, right? That's where the feature flag toggling lives. But then it seems like it's easy to devolve into spaghetti. You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns. Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this? I looked up LaunchDarkly. $299/mo for basic team support. Uhh... That's almost the cost of WeWork office space. Are feature flags really so nascent that there isn't a FOSS solution for it?
    - hoorayimhelping 9y ago
      The expectation of a bunch of weird branches making everything confusing (I had that too) ends up being unfounded. You typically only see a few places with branching logic, and people usually draw attention to it being branching logic for a feature or experiment. How and where you branch really depends on the feature. If you're A/B testing a button color, you'll deploy a different strategy than if you're trying to introduce new logic with minimal regressions. But you can plan out your strategy in both cases to minimize leaking information and disrupting user experience (and you should). >Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this? The general flow is: get status of features from source of truth. Check current feature flag's status. Do something based on that status. You can make it as complicated or simple as you like. You can have a global boolean flag for all features where it's always either true or false. You can have different flags for different users, you can have certain percentages of users have different values for flags. But it all starts from getting the value of a flag for this current request from the source of truth. Let's say you're writing a crud app on an express server that communicates via http to a database service. Your database service is rolling out v5 of their api, and they're at the last stages of their beta so they're pretty confident. You decide you want to test it by routing half of all the traffic to your crud app through v5 and the other half as normal through v4. You could lay out your code in several ways, and you'll have to figure out what works best for your model and your team. Let's assume you have a feature service you query on every request (maybe it's express middleware) to see if an experiment is enabled for the current user, and you've queried your current user against api v5, and you have a boolean variable indicating whether to show v5 or v4 and it's used in a route handler for your home route. You could change the url you hit based on the state of the experiment, provided the interface to the api is the same. const routeUrl = '/api/v4/home/' if (featureIsEnabled) { routeUrl = '/api/v5/home/' } Inside the route home route handler you could have an if/case statement that did a completely different request and response. if (featureIsEnabled) { return queryV5('home').then(respondToV5).catch(handleV5); } else { return queryV4('home').then(respondToV4).catch(handleV4); } You can also do this on the front end - a lot of feature / experiment services have front-end libraries. I prefer to do it server side.
- QuercusMax 9y ago
  There's used a ton at Google, except they're called "experiments". I believe that term comes from A/B testing, which they're still heavily used for, but they're also used for feature rollouts.
- hyperpallium 9y ago
  Dependency injection? The conditionals are then virtual (which implementation of an interface); the same controls apply to front-end code; global dependency configuration.Though all that infrastructure and boilerplate for explicit modularity is probably only worthwhile at Facebook-scale. With each feature in a separate file, there is no merging of features, and if you squint your eyes you can see each possible dependency configuration as a git branch, selecting certain files out of a repository, just at runtime. There are no file-level merges, only different files, and different configuration selections. (Of course, modifying code to use the interface in the first place is a file-level change.) BTW I've been thinking about this problem in the context of computational fluid dynamics, where you want to try different scales, fluid and container initial condition profiles and boundary conditions, as well as different discretizations, schemes and variations thereof. Basically, experimental cases (a bit like "functional test" setups).
- latencyloser 9y ago
  The project I work on at Microsoft uses feature flags. I see it with many Azure services as well. I can testify that it is absolutely if-statement hell, at least in the areas that rely on these feature flags. However, I will say I think 'lazy programming' is more the issue than the flags themselves. For example, isolating feature specific behavior to it's own impl and using an abstract class to share common behavior--that sort of basic stuff gets skipped in favor of 'time'. Thankfully these are all obscured from the user for the most part. We have a 'control panel' like thing that we used to enable/disable feature switches. It works well for its use case and let's us enable/disable with as much granularity as we want.
  - kostarelo 9y ago
    Can you give us some examples of those feature flags?
    - latencyloser 9y ago
      Hmm. I can't get into detail. But we use it for things that are mostly 'experimental' for the customer. Let's take the all-to-common TODO MVC app and say I add a caching layer between the app and the database. I would likely add a feature flag to enable or disable retrieving or storing results in the new caching layer. Or, I add a new button that lets the customer multi-select TODO tasks, that would have a feature switch, etc. The reason for this is that it lets us enable these features for customers who either insist on having them or are excited to try new features without exposing the entire customer base to the risk of these features. The big problem with this is: you end up having 'if(feature)' type statements _everywhere_ for each of those experimental buttons or extra components or whatever. It creates considerable debt that the team has to be accountable for cleaning up as those features start to stabilize.
- dahart 9y ago
  > Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there. This might be just a terminology problem. Any company that does A/B testing is using feature flags, so pretty much most of the web is using them. I'd bet more or less all programmers have them in their repos too, maybe you just call them something else, or didn't think they could be called feature flags. If you have a config file, chances are high that you have feature flags already. For tooling, I get the feeling you're thinking of feature flags as something other than your config files and code constants. But that's it: feature flags are really simple. Add a boolean const defaulted to false in your config somewhere, and wrap the code in an if statement. When you need to A/B test or release incrementally, that's when you need extra non-trivial code around features, and that's often not something someone else can provide. Turning the feature on and off requires specific knowledge of your project's stack and UI.
- guelo 9y ago
  I've used feature flags at 3 non-facebook companies.
- kccqzy 9y ago
  I think there are many different equally valid ways to use feature flags. We use feature flags to slowly ramp up the release of new features only. We have per-user feature flag settings too. We have a small team so the number of new features that need to be gradually released is very small, and won’t make the codebase a mess of `if` statements. We don’t really worry about feature flags leaking to users because when we start to use the feature flag, there is already a group of users being able to use the feature (usually employees because of dogfooding, and then gradually to a selected group of power users/VIPs). The frontend JS code is already minified beyond recognition (we use Google Closure compiler in advanced mode) so that’s not really a concern. Since feature flags are per-user, you can think of them as a column of the users table in the database.
- kcorbitt 9y ago
  At Google every significant feature in our codebase was behind a feature flag. Pro is that you can roll out features slowly and do a statistical analysis of their impact on user behavior in a well-controlled way. Also, you can turn off a broken feature really fast without having to wait for a new deploy (especially important with native mobile apps). The big downside was that it significantly impacted developer velocity. You have to test that everything still works with/without your feature enabled. Also, sometimes supporting the old and new way of doing things simultaneously requires a nasty hack, and when you go back in to pull out the old code after a successful deploy you don't actually bother to rearchitect things the way you ought to. For Google they were probably inevitable, but for a small to mid-sized company I would avoid them.
- baylisscg 9y ago
  Feature flags _are_ widely used. They generally manifest as commented out code. Like many features that migrate down from these massive codebases it's a common behaviour thats been systematised.
- krzyk 9y ago
  I've worked at Sabre some time ago and we used feature flags intensively. Feature was enabled/disabled using our own configuration server (which was used to configure many other things) - it was basically a dynamic file with booleans, that we could turn on/off. After the feature was released we removed the if statement from the code.
damnyou 9y ago
This is called trunk-based development, and it is the only development model that scales to thousands of engineers in one repository.
- dorian-graph 9y ago
  While I do like this approach, it can cause some pains though. It's often a race to get your (approved) merge- or pull- request in. If you miss out, then you rebase, and try again. Perhaps if your pipeline is super fast, it isn't an issue. Another issue is that there still needs to be coordination outside of version control/CI pipeline to ensure that things are put in the 'right order', for the times when that is important.
  - damnyou 9y ago
    Developers rarely push directly to master in large organizations. Instead, a CI pipeline verifies your code passes tests and then pushes to master. There are trade-offs here around "semantic merge conflicts", but they happen rarely in practice. To avoid making this disruptive to developers, you can have another branch that marks the latest commit to pass tests. This branch should always be an ancestor of master and should only be updated by your CI pipeline.
    - dorian-graph 9y ago
      > Developers rarely push directly to master in large organizations. Instead, a CI pipeline verifies your code passes tests and then pushes to master. Yup, that's what I was referring to with my 'merge- and pull- request' line. Interesting idea about having another branch.
  - nebabyte 9y ago
    And why not. Rebasing should be much more straightforward/common. I remember at one point I had a 'magic' command that just did a rebase the way I expected and allowed me to merge changes in easily; it quickly made its way through the group I was working with.
- 9y ago
radicalbyte 9y ago
This. Anything else is insanity.
ryanbrunner 9y ago
I think from the perspective of the central repository, this is a sane approach. Locally though, you should still be routinely using (short-lived) feature branches. Working directly off master on your local machine can make things annoying / complicated when you need to move away from your work for a moment (say to fix a bug) and don't have a good representation of "what's currently on production". Also, I think this approach necessitates that you have some form of continuous deployment, or strong assurances that master is almost always on production. If, for example, you have a fixed 2-week cycle for production releases, "just work off master" will be a source of pain when you inevitably have to fix a bug on production without rolling out the other commits that are there for the "next release". That might not be an optimal way of developing, but it's about more than just how you use git - it's really more of a delivery question.
- bbatha 9y ago
  Depending on the frequency of your deploys you can simply add release branches or tags to the “work on master” model. Any decent deployment system can deploy from an arbitrary arbitrary ref, plus release history in your git log is very useful. The point here is to eliminate long lived feature branches that require their own maintenance.
  - geezerjay 9y ago
    > Depending on the frequency of your deploys you can simply add release branches or tags to the “work on master” model. ...and then you get the git workflow described in the blog post. It only takes a single regression to learn the importance of separating the development branch from the main branch.
- skrebbel 9y ago
  A hotfix branch fixes this. You can still do trunk based development for the rest, and cherry pick the bugfixes into such a hotfix branch (branches off from the latest commit that was released).
  - TheRealWatson 9y ago
trjordan 9y ago
I like feature flags, but it's crazy to suggest that they totally mitigate your risk. Some features will end up with more lines of code hiding the feature than actually implementing it. After it's released, you then have to unwind all that code. That code churn isn't trivial, and "tests pass, ship it" isn't really a good model if you're shipping blocks of work that you don't want to expose to customers. If your releases processes are perfect, then it's OK, but CI tends to treat release as "if the tests pass, it's perfect" and ships it to all customers. At the very least, you need to be able to compare releases that "don't change anything" to make sure you're not breaking something because some edge of your feature didn't get hidden behind the flag.
MaulingMonkey 9y ago
Here's the caveats I have with that, although I do trend towards only having a master central branch whenever possible: 1) Small, incremental commits tend to be way more reviewable than big "finished" blobs of work. Local/small branches prevent me from getting blocked on waiting for review, and allows me to continue working, although you don't exactly need a "branching model" for it. 2) Tests/CI are insufficient. For games, you'll want QA hammering on a release for days, weeks, or months with only the most conservative changes applied, in an effort to shake out any remaining weird heisenbugs. To do otherwise invites the specter of failing certification, delaying your release, or burning some pretty bad bugs to hundreds of thousands of physical disks. Do you, effectively, shut down the studio when you've dialed up the stability demands that much - or do you branch so work can continue? The worst case consequences are no longer quite so bad as burning bugs into unpatchable ROMs, but they can still be pretty bad. > The time between code being written and a bug being seen can be reduced to minutes and hours, making finding the root cause a breeze. This is, unfortunately, merely the ideal happy path. Great when it happens, but fails to account for the worst case. It can take weeks, months, to find the heisenbugs that slip through CI/tests/initial QA. If you have a CI/testing setup that can catch, say, a title exit(3)ing when the charm bar is opened for more than 10 seconds, but only on the main menu and without a debugger attached - without writing a test to catch that extremely specific edge case once you have the benefit of hindsight - I'm begging you to share! Just nailing down the exact repro steps took days. When you have no relevant callstack (even when you do eventually figure out how to dump the exit(3) event), no relevant logs, the fundamental bug resides in third party code you don't have the source code to, and fully rebuilding takes hours because you're on a large C++ codebase with even larger asset building requirements - well, for me, it took me a week or two to root cause (or more accurately, get fed up to the point that I spent a day manually bisecting version history to track down a totally unrelated and innocuous looking changelist that ultimately made us hit the bug.) In the weeks of turnaround time fixing such a bug, a sufficiently active dev branch will have acquired another. A release branch won't. Bam, branching model. 3) Plenty of codebase-wide refactoring is hard or impossible to feature-flag.
- sillysaurus3 9y ago
  Tests/CI are insufficient. For games, you'll want QA hammering on a release for days, weeks, or months with only the most conservative changes applied, in an effort to shake out any remaining weird heisenbugs. To do otherwise invites the specter of failing certification, delaying your release, or burning some pretty bad bugs to hundreds of thousands of physical disks. Eh.. Game studios aren't generally still in the dark ages, are they? Physical disks went the way of the dodo. (Every game now basically requires updates on first use, right? So even if consumers get a disc[o ball], they won't really be affected by a bug burned to it.) But yes, solid points.
viraptor 9y ago
> and never merge/push a change that won't pass tests/CI. This can be enforced as well. For example using Gerrit + some CI, you can allow people to submit a branch to be merged, but it's up to the CI to merge it if it passes all checks. There's no "aren't allowed" anymore. (Which is great because people will make mistakes) One thing to keep in mind though is you're describing a system that works for a continuously deployed web service for example. Once you have a product with versions, you need to have a system for previous releases, backporting patches, security/point updates, etc. It's a completely different game at that point.
nebabyte 9y ago
> I've got a much nicer branching model- try not to have one Ah, like my much nicer version control system ;)
dboreham 9y ago
Exactly. I've never understood the obsession with minor branching. Actually now we never ship software in a box major branches typically aren't needed either. The insanity goes back way before git but git seems to have driven it to "11".
vermooten 9y ago
mabbo you are right, it's how we do it and it's hyper-efficient. Last place I worked used OP's model it worked ok but wasn't as efficient.
dahart 9y ago
Personally, I wouldn't argue for one or the other being nicer, they each have strong use cases at different times. I like to use feature flags for: 1- features that need to be A/B tested, 2- huge features that dev lasts for more than 1-2 weeks, 3- features that take more than a small handful of people to either write or test. I like to use branches for: 1- code that definitely should not be inflicted on other teams until ready, 2- features that take between 3 days and a week or two, relatively short cycle, and 3- features that need more than one person and/or more than one commit to finish, but less than maybe half a dozen. Assuming it doesn't impact the code review process, I like to commit to master directly, without a feature flag, for: 1- any small features contained in a single commit written by a single person, and 2- hot patches and bug fixes that roll in after merge.
nerdwaller 9y ago
I think it really depends on what you’re developing and delivering. Master based development is great for end-user deliverables (web apps being the big one that comes to mind, many libraries could do this too). But if you’re writing a language or LTS releases I could see a pretty good case for a branching model like git-flow. My experience with a work using gitflow is similar to yours, it was very messy and hard to manage. It was even worse that those driving it didn’t really know much about git so 95% of the commit messages were “Merged X into Y...” instead of using a more linear history.
- kostarelo 9y ago
  > It was even worse that those driving it didn’t really know much about git so 95% of the commit messages were “Merged X into Y...” instead of using a more linear history. How so? Why merge commits are bad? Are you suggesting of rebasing each topic branch or commiting directly to the master?
  - damnyou 9y ago
    Merge commits are bad because they add unnecessary complexity. Your software is already complex enough. Why add complexity to your commit logs and your overall development process? A linear history is easy to manage and easy to understand. It might be slightly worse for people committing code, but developers read several orders of magnitude more code than they write. Always optimize for reading code over writing it.
nikolasavic 9y ago
You're not the only who thinks this is a great idea. https://www.thoughtworks.com/insights/blog/enabling-trunk-based-development-deployment-pipelines https://www.thoughtworks.com/insights/blog/enabling-trunk-ba...
webo 9y ago
I've been trying to think more about this model. How do people perform code reviews with incomplete features? I feel like you lose context as you review incremental commits that span across multiple days.
geezerjay 9y ago
> I've got a much nicer branching model- try not to have one. Everyone works off master, and you aren't allowed to check in code that won't run in production. Hide unfinished features behind feature flags, and never merge/push a change that won't pass tests/CI. You've just described a naive implementation of the Git branching model described in the article which fails to meet some basic real-world requirements experienced on all software development projects. It matches the policy of merging changes from the development branch onto the master branch but lacking any support for very basic aspects of software development such as code auditing, versioning, and continuous integration. It's silly to let everyone work off master if master is supposed to remain stable. If master is supposed to host stable versions, someone needs to be responsible for keeping it stable. If someone is assigned the task of ensuring the master branch is kept stable then he needs to control what gets into master and what doesn't. If someone needs to control what gets into master then other developers need to commit their changes to some other branch. Add parallel releases and couple of levels of these real-world everyday aspects of software development and you get exactly the branching model described by Vincent Driessen in his blog post.
- chris_wot 9y ago
  Agreed - you can have compiling code that passes all tests but which still needs someone to review. Or a feature that you are working on that you want to merge in later.
lmilcin 9y ago
I also agree. Realizing there can't be perfect model, I started promoting working directly with the master. This requires to work out the problems in advance instead of letting them wait until merge time, but this . Having one branch means every team member is more or less aware of what is happening. It is much easier and it makes much more sense to set up automated pipelines if you have just one branch. Having everybody work on the same branch seems to focus people on gradual improving the codebase instead of making large rewrites.

by mabbo 9y ago

sillysaurus3 9y ago

Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there. Some concrete questions: - How do you prevent your codebase from becoming if-statement spaghetti? - How do you prevent new features from being leaked to the user? They'll see the new features in the front end source code. At many companies this isn't an acceptable tradeoff. So do you preprocess the release code and strip out the disabled feature flags? With what? - What do you use to control feature flags? Just a json file filled with `"foo-bar-feature": true/false`, or something more sophisticated like a control panel that you can use to say "10% of our users will see this feature"?

NicoJuicy 9y ago
Who says no one is embracing them? There are tons of tools available if you check for it. But you should google for "feature toggle" libraries and not feature flags
mabbo 9y ago
> How do you prevent your codebase from becoming if-statement spaghetti? A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch, and launch fully or remove the feature; consider 'hiding' the ifs behind factories that build the objects that implement the different logic; also, if you have 20 features is development for the same area of your codebase, worry- you may be trying too much! > How do you prevent new features from being leaked to the user? I haven't had to worry about this very often just based on my projects, but there are strategies. You could be sending the user a different version of the js/html based on the feature flags (preprocessor as you said). Haven't had to do that, but woah that would be a fun little project. > What do you use to control feature flags? My favorite implementation was an S3 json file that effectively encoded a decision tree based on variables used. In code, we could say "here are the five variables about this request, feature object are you enabled?". By modifying that json file, you could change the features state at run time. (Note: this was not perfectly implemented/designed by me and caused a few issues when we first used it. Oops. But you can do very quick solutions too, read a file or make an object that decides based on non-dynamic logic.
- jasode 9y ago
  >A valid point- if you're not careful that can happen. Key things: remove those ifs after a launch I think a big factor that's unspoken about "trunk-based-master" development is that it fits better with "website apps" such as Facebook and Etsy. The "live website" can be thought of as a "single executable" and trunk-based mental model maps well to that. The feature-flags as a replacement for branches doesn't result in an unusable combinatorial explosion. There are minimal # of "alternate universes" of Facebook... maybe a "Facebook for internal engineers" etc. You can tame that with feature-flags. However, for corporate development where the output is "exe" files... e.g. version 2.x of product has dependencies on a old version of a library and a version 3.x uses a totally different architecture with different conflicting dependencies... you can't model that code evolution in a clean manner with feature-flag "if" statements. If you move the "if" statements outside of the code and into preprocessors (#ifdef) or the "build" system such as cmake, you've just shifted the problem around instead of solving it. Branches instead of feature-flags are cleaner and more sane for certain domains.
  - mabbo 9y ago
    Fortunately, exe development is becoming a small subset of software these days.
hoorayimhelping 9y ago
>Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them I worked at Etsy from early 2012 to late 2015 and I used feature flags every day I was there. >How do you prevent your codebase from becoming if-statement spaghetti? You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go. >How do you prevent new features from being leaked to the user? You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns. >What do you use to control feature flags? You can use a service like LaunchDarkly or Optimizely or you can write your own service, or you can have a file in memory. Usually it starts simple with boolean toggles and evolves into a ramp up system that let's you selectively target users. The important part is that you need to be able to change features without redeploying code.
- sillysaurus3 9y ago
  You remove the feature flag checking code when you turn off or turn on the feature. It's an extra deploy, but it needs to be part of the prod/eng process. This is the most salient issue of feature flagging in my experience and something you need to get ahead of from the get-go. I don't understand. I was asking about the development source code, not the code delivered to the user. The dev source code isn't stripped of if-statements, right? That's where the feature flag toggling lives. But then it seems like it's easy to devolve into spaghetti. You branch at the server/request level, not the client level, so there is no set of features that gets displayed, it's just what the server returns. Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this? I looked up LaunchDarkly. $299/mo for basic team support. Uhh... That's almost the cost of WeWork office space. Are feature flags really so nascent that there isn't a FOSS solution for it?
  - hoorayimhelping 9y ago
    The expectation of a bunch of weird branches making everything confusing (I had that too) ends up being unfounded. You typically only see a few places with branching logic, and people usually draw attention to it being branching logic for a feature or experiment. How and where you branch really depends on the feature. If you're A/B testing a button color, you'll deploy a different strategy than if you're trying to introduce new logic with minimal regressions. But you can plan out your strategy in both cases to minimize leaking information and disrupting user experience (and you should). >Ok, but how? I'm not playing dumb. Assume an express server. You want to put a feature flag into your codebase. What is step #1 (and #2 and ... #N) to achieve this? The general flow is: get status of features from source of truth. Check current feature flag's status. Do something based on that status. You can make it as complicated or simple as you like. You can have a global boolean flag for all features where it's always either true or false. You can have different flags for different users, you can have certain percentages of users have different values for flags. But it all starts from getting the value of a flag for this current request from the source of truth. Let's say you're writing a crud app on an express server that communicates via http to a database service. Your database service is rolling out v5 of their api, and they're at the last stages of their beta so they're pretty confident. You decide you want to test it by routing half of all the traffic to your crud app through v5 and the other half as normal through v4. You could lay out your code in several ways, and you'll have to figure out what works best for your model and your team. Let's assume you have a feature service you query on every request (maybe it's express middleware) to see if an experiment is enabled for the current user, and you've queried your current user against api v5, and you have a boolean variable indicating whether to show v5 or v4 and it's used in a route handler for your home route. You could change the url you hit based on the state of the experiment, provided the interface to the api is the same. const routeUrl = '/api/v4/home/' if (featureIsEnabled) { routeUrl = '/api/v5/home/' } Inside the route home route handler you could have an if/case statement that did a completely different request and response. if (featureIsEnabled) { return queryV5('home').then(respondToV5).catch(handleV5); } else { return queryV4('home').then(respondToV4).catch(handleV4); } You can also do this on the front end - a lot of feature / experiment services have front-end libraries. I prefer to do it server side.
QuercusMax 9y ago
There's used a ton at Google, except they're called "experiments". I believe that term comes from A/B testing, which they're still heavily used for, but they're also used for feature rollouts.
hyperpallium 9y ago
Dependency injection? The conditionals are then virtual (which implementation of an interface); the same controls apply to front-end code; global dependency configuration.Though all that infrastructure and boilerplate for explicit modularity is probably only worthwhile at Facebook-scale. With each feature in a separate file, there is no merging of features, and if you squint your eyes you can see each possible dependency configuration as a git branch, selecting certain files out of a repository, just at runtime. There are no file-level merges, only different files, and different configuration selections. (Of course, modifying code to use the interface in the first place is a file-level change.) BTW I've been thinking about this problem in the context of computational fluid dynamics, where you want to try different scales, fluid and container initial condition profiles and boundary conditions, as well as different discretizations, schemes and variations thereof. Basically, experimental cases (a bit like "functional test" setups).
latencyloser 9y ago
The project I work on at Microsoft uses feature flags. I see it with many Azure services as well. I can testify that it is absolutely if-statement hell, at least in the areas that rely on these feature flags. However, I will say I think 'lazy programming' is more the issue than the flags themselves. For example, isolating feature specific behavior to it's own impl and using an abstract class to share common behavior--that sort of basic stuff gets skipped in favor of 'time'. Thankfully these are all obscured from the user for the most part. We have a 'control panel' like thing that we used to enable/disable feature switches. It works well for its use case and let's us enable/disable with as much granularity as we want.
- kostarelo 9y ago
  Can you give us some examples of those feature flags?
  - latencyloser 9y ago
    Hmm. I can't get into detail. But we use it for things that are mostly 'experimental' for the customer. Let's take the all-to-common TODO MVC app and say I add a caching layer between the app and the database. I would likely add a feature flag to enable or disable retrieving or storing results in the new caching layer. Or, I add a new button that lets the customer multi-select TODO tasks, that would have a feature switch, etc. The reason for this is that it lets us enable these features for customers who either insist on having them or are excited to try new features without exposing the entire customer base to the risk of these features. The big problem with this is: you end up having 'if(feature)' type statements _everywhere_ for each of those experimental buttons or extra components or whatever. It creates considerable debt that the team has to be accountable for cleaning up as those features start to stabilize.
dahart 9y ago
> Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. The tooling just isn't there. This might be just a terminology problem. Any company that does A/B testing is using feature flags, so pretty much most of the web is using them. I'd bet more or less all programmers have them in their repos too, maybe you just call them something else, or didn't think they could be called feature flags. If you have a config file, chances are high that you have feature flags already. For tooling, I get the feeling you're thinking of feature flags as something other than your config files and code constants. But that's it: feature flags are really simple. Add a boolean const defaulted to false in your config somewhere, and wrap the code in an if statement. When you need to A/B test or release incrementally, that's when you need extra non-trivial code around features, and that's often not something someone else can provide. Turning the feature on and off requires specific knowledge of your project's stack and UI.
guelo 9y ago
I've used feature flags at 3 non-facebook companies.
kccqzy 9y ago
I think there are many different equally valid ways to use feature flags. We use feature flags to slowly ramp up the release of new features only. We have per-user feature flag settings too. We have a small team so the number of new features that need to be gradually released is very small, and won’t make the codebase a mess of `if` statements. We don’t really worry about feature flags leaking to users because when we start to use the feature flag, there is already a group of users being able to use the feature (usually employees because of dogfooding, and then gradually to a selected group of power users/VIPs). The frontend JS code is already minified beyond recognition (we use Google Closure compiler in advanced mode) so that’s not really a concern. Since feature flags are per-user, you can think of them as a column of the users table in the database.
kcorbitt 9y ago
At Google every significant feature in our codebase was behind a feature flag. Pro is that you can roll out features slowly and do a statistical analysis of their impact on user behavior in a well-controlled way. Also, you can turn off a broken feature really fast without having to wait for a new deploy (especially important with native mobile apps). The big downside was that it significantly impacted developer velocity. You have to test that everything still works with/without your feature enabled. Also, sometimes supporting the old and new way of doing things simultaneously requires a nasty hack, and when you go back in to pull out the old code after a successful deploy you don't actually bother to rearchitect things the way you ought to. For Google they were probably inevitable, but for a small to mid-sized company I would avoid them.
baylisscg 9y ago
Feature flags _are_ widely used. They generally manifest as commented out code. Like many features that migrate down from these massive codebases it's a common behaviour thats been systematised.
krzyk 9y ago
I've worked at Sabre some time ago and we used feature flags intensively. Feature was enabled/disabled using our own configuration server (which was used to configure many other things) - it was basically a dynamic file with booleans, that we could turn on/off. After the feature was released we removed the if statement from the code.

damnyou 9y ago

This is called trunk-based development, and it is the only development model that scales to thousands of engineers in one repository.

dorian-graph 9y ago
While I do like this approach, it can cause some pains though. It's often a race to get your (approved) merge- or pull- request in. If you miss out, then you rebase, and try again. Perhaps if your pipeline is super fast, it isn't an issue. Another issue is that there still needs to be coordination outside of version control/CI pipeline to ensure that things are put in the 'right order', for the times when that is important.
- damnyou 9y ago
  Developers rarely push directly to master in large organizations. Instead, a CI pipeline verifies your code passes tests and then pushes to master. There are trade-offs here around "semantic merge conflicts", but they happen rarely in practice. To avoid making this disruptive to developers, you can have another branch that marks the latest commit to pass tests. This branch should always be an ancestor of master and should only be updated by your CI pipeline.
  - dorian-graph 9y ago
    > Developers rarely push directly to master in large organizations. Instead, a CI pipeline verifies your code passes tests and then pushes to master. Yup, that's what I was referring to with my 'merge- and pull- request' line. Interesting idea about having another branch.
- nebabyte 9y ago
  And why not. Rebasing should be much more straightforward/common. I remember at one point I had a 'magic' command that just did a rebase the way I expected and allowed me to merge changes in easily; it quickly made its way through the group I was working with.
9y ago

radicalbyte 9y ago

This. Anything else is insanity.

ryanbrunner 9y ago

I think from the perspective of the central repository, this is a sane approach. Locally though, you should still be routinely using (short-lived) feature branches. Working directly off master on your local machine can make things annoying / complicated when you need to move away from your work for a moment (say to fix a bug) and don't have a good representation of "what's currently on production". Also, I think this approach necessitates that you have some form of continuous deployment, or strong assurances that master is almost always on production. If, for example, you have a fixed 2-week cycle for production releases, "just work off master" will be a source of pain when you inevitably have to fix a bug on production without rolling out the other commits that are there for the "next release". That might not be an optimal way of developing, but it's about more than just how you use git - it's really more of a delivery question.

bbatha 9y ago
Depending on the frequency of your deploys you can simply add release branches or tags to the “work on master” model. Any decent deployment system can deploy from an arbitrary arbitrary ref, plus release history in your git log is very useful. The point here is to eliminate long lived feature branches that require their own maintenance.
- geezerjay 9y ago
  > Depending on the frequency of your deploys you can simply add release branches or tags to the “work on master” model. ...and then you get the git workflow described in the blog post. It only takes a single regression to learn the importance of separating the development branch from the main branch.
skrebbel 9y ago
A hotfix branch fixes this. You can still do trunk based development for the rest, and cherry pick the bugfixes into such a hotfix branch (branches off from the latest commit that was released).
- TheRealWatson 9y ago

trjordan 9y ago

I like feature flags, but it's crazy to suggest that they totally mitigate your risk. Some features will end up with more lines of code hiding the feature than actually implementing it. After it's released, you then have to unwind all that code. That code churn isn't trivial, and "tests pass, ship it" isn't really a good model if you're shipping blocks of work that you don't want to expose to customers. If your releases processes are perfect, then it's OK, but CI tends to treat release as "if the tests pass, it's perfect" and ships it to all customers. At the very least, you need to be able to compare releases that "don't change anything" to make sure you're not breaking something because some edge of your feature didn't get hidden behind the flag.

MaulingMonkey 9y ago

Here's the caveats I have with that, although I do trend towards only having a master central branch whenever possible: 1) Small, incremental commits tend to be way more reviewable than big "finished" blobs of work. Local/small branches prevent me from getting blocked on waiting for review, and allows me to continue working, although you don't exactly need a "branching model" for it. 2) Tests/CI are insufficient. For games, you'll want QA hammering on a release for days, weeks, or months with only the most conservative changes applied, in an effort to shake out any remaining weird heisenbugs. To do otherwise invites the specter of failing certification, delaying your release, or burning some pretty bad bugs to hundreds of thousands of physical disks. Do you, effectively, shut down the studio when you've dialed up the stability demands that much - or do you branch so work can continue? The worst case consequences are no longer quite so bad as burning bugs into unpatchable ROMs, but they can still be pretty bad. > The time between code being written and a bug being seen can be reduced to minutes and hours, making finding the root cause a breeze. This is, unfortunately, merely the ideal happy path. Great when it happens, but fails to account for the worst case. It can take weeks, months, to find the heisenbugs that slip through CI/tests/initial QA. If you have a CI/testing setup that can catch, say, a title exit(3)ing when the charm bar is opened for more than 10 seconds, but only on the main menu and without a debugger attached - without writing a test to catch that extremely specific edge case once you have the benefit of hindsight - I'm begging you to share! Just nailing down the exact repro steps took days. When you have no relevant callstack (even when you do eventually figure out how to dump the exit(3) event), no relevant logs, the fundamental bug resides in third party code you don't have the source code to, and fully rebuilding takes hours because you're on a large C++ codebase with even larger asset building requirements - well, for me, it took me a week or two to root cause (or more accurately, get fed up to the point that I spent a day manually bisecting version history to track down a totally unrelated and innocuous looking changelist that ultimately made us hit the bug.) In the weeks of turnaround time fixing such a bug, a sufficiently active dev branch will have acquired another. A release branch won't. Bam, branching model. 3) Plenty of codebase-wide refactoring is hard or impossible to feature-flag.

sillysaurus3 9y ago
Tests/CI are insufficient. For games, you'll want QA hammering on a release for days, weeks, or months with only the most conservative changes applied, in an effort to shake out any remaining weird heisenbugs. To do otherwise invites the specter of failing certification, delaying your release, or burning some pretty bad bugs to hundreds of thousands of physical disks. Eh.. Game studios aren't generally still in the dark ages, are they? Physical disks went the way of the dodo. (Every game now basically requires updates on first use, right? So even if consumers get a disc[o ball], they won't really be affected by a bug burned to it.) But yes, solid points.

viraptor 9y ago

> and never merge/push a change that won't pass tests/CI. This can be enforced as well. For example using Gerrit + some CI, you can allow people to submit a branch to be merged, but it's up to the CI to merge it if it passes all checks. There's no "aren't allowed" anymore. (Which is great because people will make mistakes) One thing to keep in mind though is you're describing a system that works for a continuously deployed web service for example. Once you have a product with versions, you need to have a system for previous releases, backporting patches, security/point updates, etc. It's a completely different game at that point.

nebabyte 9y ago

> I've got a much nicer branching model- try not to have one Ah, like my much nicer version control system ;)

dboreham 9y ago

Exactly. I've never understood the obsession with minor branching. Actually now we never ship software in a box major branches typically aren't needed either. The insanity goes back way before git but git seems to have driven it to "11".

vermooten 9y ago

mabbo you are right, it's how we do it and it's hyper-efficient. Last place I worked used OP's model it worked ok but wasn't as efficient.

dahart 9y ago

Personally, I wouldn't argue for one or the other being nicer, they each have strong use cases at different times. I like to use feature flags for: 1- features that need to be A/B tested, 2- huge features that dev lasts for more than 1-2 weeks, 3- features that take more than a small handful of people to either write or test. I like to use branches for: 1- code that definitely should not be inflicted on other teams until ready, 2- features that take between 3 days and a week or two, relatively short cycle, and 3- features that need more than one person and/or more than one commit to finish, but less than maybe half a dozen. Assuming it doesn't impact the code review process, I like to commit to master directly, without a feature flag, for: 1- any small features contained in a single commit written by a single person, and 2- hot patches and bug fixes that roll in after merge.

nerdwaller 9y ago

I think it really depends on what you’re developing and delivering. Master based development is great for end-user deliverables (web apps being the big one that comes to mind, many libraries could do this too). But if you’re writing a language or LTS releases I could see a pretty good case for a branching model like git-flow. My experience with a work using gitflow is similar to yours, it was very messy and hard to manage. It was even worse that those driving it didn’t really know much about git so 95% of the commit messages were “Merged X into Y...” instead of using a more linear history.

kostarelo 9y ago
> It was even worse that those driving it didn’t really know much about git so 95% of the commit messages were “Merged X into Y...” instead of using a more linear history. How so? Why merge commits are bad? Are you suggesting of rebasing each topic branch or commiting directly to the master?
- damnyou 9y ago
  Merge commits are bad because they add unnecessary complexity. Your software is already complex enough. Why add complexity to your commit logs and your overall development process? A linear history is easy to manage and easy to understand. It might be slightly worse for people committing code, but developers read several orders of magnitude more code than they write. Always optimize for reading code over writing it.

nikolasavic 9y ago

You're not the only who thinks this is a great idea. https://www.thoughtworks.com/insights/blog/enabling-trunk-based-development-deployment-pipelines https://www.thoughtworks.com/insights/blog/enabling-trunk-ba...

webo 9y ago

I've been trying to think more about this model. How do people perform code reviews with incomplete features? I feel like you lose context as you review incremental commits that span across multiple days.

geezerjay 9y ago

> I've got a much nicer branching model- try not to have one. Everyone works off master, and you aren't allowed to check in code that won't run in production. Hide unfinished features behind feature flags, and never merge/push a change that won't pass tests/CI. You've just described a naive implementation of the Git branching model described in the article which fails to meet some basic real-world requirements experienced on all software development projects. It matches the policy of merging changes from the development branch onto the master branch but lacking any support for very basic aspects of software development such as code auditing, versioning, and continuous integration. It's silly to let everyone work off master if master is supposed to remain stable. If master is supposed to host stable versions, someone needs to be responsible for keeping it stable. If someone is assigned the task of ensuring the master branch is kept stable then he needs to control what gets into master and what doesn't. If someone needs to control what gets into master then other developers need to commit their changes to some other branch. Add parallel releases and couple of levels of these real-world everyday aspects of software development and you get exactly the branching model described by Vincent Driessen in his blog post.

chris_wot 9y ago
Agreed - you can have compiling code that passes all tests but which still needs someone to review. Or a feature that you are working on that you want to merge in later.

lmilcin 9y ago

I also agree. Realizing there can't be perfect model, I started promoting working directly with the master. This requires to work out the problems in advance instead of letting them wait until merge time, but this . Having one branch means every team member is more or less aware of what is happening. It is much easier and it makes much more sense to set up automated pipelines if you have just one branch. Having everybody work on the same branch seems to focus people on gradual improving the codebase instead of making large rewrites.