12 ms·

When deployments are easy, code becomes simpler

189 points by recroad 3y ago

KevinMS 3y ago
I'm confused. This is standard practice for CI/CD, but nobody is mentioning it. Is CI/CD that obscure?
dmoy 3y ago
> Is this bad because I now have to do a deployment to enable the feature? ... I would argue as long as your deployments are easy, this is the better way to do things because it reduces the complexity of integrating with a third-party tool. Shoot, even when it's all first party tooling, I prefer a release-flags-and-binary as an atomic unit. If the flags and binary are going out as a single push, it simplifies a lot of things: * reproducible state for given time with only one thing to track (running version), instead of two or more things (binary version, flag configs version) * corollary: rollbacks are much more simple, because it's just one thing to rollback - the code+config+everything as an atomic unit, not "did we need to roll back the flags, or the binary, or both?" (Corollary to corollary: removes the awful problem of "oh shit the code and/or flags/configs wasn't backwards/forwards compatible" where you shoot yourself in the foot while doing a rollback * removals and cleanups are easier: can remove flags, config bits, and code all at once, instead of having to do a careful dance of "set code behavior to default-on and remove dependency on flag, the wait for full deployment, then make sure everyone is okay with not rolling back after a given point, then remove flags and config logic" * depending on your tooling, diffs vs prod during code review get cleaner
- ljm 3y ago
  The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.
  - mcny 3y ago
    I always say this - there is no such thing as a rollback. Anyone who pretends you can simply "roll back" a stateful application is out of their mind. If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.
    - perrygeo 3y ago
      The system can only move forwards. Just like a `git revert`, it's a new commit to an immutable history. This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.
      - OJFord 3y ago
        In what sense is 'downgrade' not just that prepared ahead of time? Or do you just not like the name? Of course usually it would be followed by something that fixes it 'properly' - applying the original 'upgrade' again but with some correction, but typically that'll take longer than reverting to the previous version, and given it's gone wrong you'll probably want to take longer with it, test more, restore confidence.
        perrygeo 3y ago
        downgrade implies that such a thing is possible. In a stateful system though, you cannot take back that which you have committed. It's quite literally impossible by definition.
        OJFord 3y ago
        No it isn't? Forward migration: add date column, set = date(day, month, year) for existing records, drop is valid date constraint, drop day/month/year columns Down migration: add day/month/year columns, add is valid date constraint, set day/month/year = appropriate datepart of date, drop date column for trivial example.
  - chaxor 3y ago
    If a git commit hash was directly tied to a data hash at that state (IPFS), that would be trivial.
    - tbrownaw 3y ago
      Or have the SAN make a snapshot of your db before the deploy, no fancy solution-looking-for-a-problem tech needed. Completely reverting all the state to some point in the past is easy and have been a solved problem for quite a while. The actual problem is undoing states changes related to the code change, while not losing state changes related to normal system activity. If I add some new fields to how my web forum records posts, and then find out that it's eating every tenth post and need to revert, it'd be good to not lose and posts made in the meantime.
      - chaxor 3y ago
        Good point, this system is much more useful for scientists and ML researchers, etc. wherein the code may generate datasets over several weeks or months of compute time, but produce some output that should be directly tied to the code. You're right though. Systems in the social media space dealing with users appending to a database typically don't have this requirement to be easily reproducible, so the local snapshots are the right choice.
    - baq 3y ago
      And useless. Unless customers don’t care about their data.
      - chaxor 3y ago
        I don't understand your statement. If a git clone of a specific commit also pulls the data that is generated from that commit hash via IPFS simultaneously, then you can always have the data at whatever state you choose. How might this have a negative effect on conserving the data? Do you mean that there are additional structures required to manage keys if the data is encrypted?
    - rjmunro 3y ago
  - dmoy 3y ago
    Sure, that is still a problem. But that problem exists whether or not your flags/configs are pushed atomically with your code/binary changes. Having flags push separately adds another problem on top of data corruption.
  - mrweasel 3y ago
    One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a feature flags (in the database), ensuring that version 2 could run, without using the new database schema. Once they where confident that everything was still running on version 2 of the code and database they'd enable the new feature. Later the feature flag could be migrated from the database to a properties file, saving the database lookup. I wouldn't necessarily call this approach simple, but it was incredibly safe and rollbacks was always a none event.
    - steffres 3y ago
      That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?
      - jaimebuelta 3y ago
        You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) both columns exist, but only the old one is used 2. Deploy new code that works with both columns, reading from old and writing new and old 3. Deploy data migration (DB migration) that ensures old and new columns has the same values (to ensure data consistency). At this point, there are no “old column only” writes by the code deployed in previous step 4. Deploy new code using only new column. Old column is deprecated 5. Delete old column At any given point, code versions N (current) and N-1 (previous) are compatible with the DB. Any change on the DB is done in advance in a backwards compatible way.
        3y ago
- gonzan 3y ago
  I agree with all this and it’s how I do it for web development but the time Apple/Google take to approve apps makes doing this for mobile apps quite risky since it’s hard to rollback. I guess it goes to the point of the author that hard deployments on mobile makes mobile development harder
teeray 3y ago
Anyone have advice for teams without control over when their software is deployed by the people actually running it? Say you have a product released to a third party and they may or may not take releases and may be arbitrarily delayed in deploying those they choose to take.
- GrinningFool 3y ago
  You're in a for a long tail of pain if you have significant uptake. To avoid this, you will need have a clear policy on what you're willing to support (eg versions up to 1 year old, or "major version - 1", etc.), and stick to it. Otherwise, people will expect you to support all of it, forever.
  - hinkley 3y ago
    Bug fix releases never get deployed. If you want people to keep up to date with your releases, you either have to provide an automatic update facility that they have to manually turn off (ugh, too hard), or you have to dangle desirable features as carrots in front of them, so that upgrading is exciting instead of a social obligation.
    - jcrben 3y ago
      My experience is kinda the opposite. Big companies get into certain workflows and don't want to change, altho they are happy to take bugfixes. This is also where I'm at with much of the software I rely upon, some of which I pull at my own whim.
- GauntletWizard 3y ago
  Give yourself as much leverage as you can get over their deployment cycles. Use both carrot and stick - Include features they can't live without with the hardest of deployment maintenance burdens, and make sure you have tight adherence to your software lifecycle - Every question about 1.0 should be answered with "We don't know, 1.0 is out of support" from the second that it's EOL date passes. Include the currently running version in every logfile, if not every log statement. Make it clear to you where they are and run your own run-behind that upgrades only after they do. This will be painful, but it will save you overall time.
bsima 3y ago
> I did an inventory of my recent feature flags and realized that about 80% of them aren’t there to roll things out to specific populations, or do any sort of A/B testing, but to hide unfinished code. Maybe it’s just me but having unfinished, dead, or scratch code in a production codebase really annoys me. Either finish your work or delete the unneeded code. More than a few times I’ve sunk time out of my day into investigating some code path only to realize it’s completely unused.
- deleted 3y ago
  [deleted]
- erik_seaberg 3y ago
  “Unfinished” can mean abandoned, but it can also mean half-baked code that has been pushed to master even though it’s still actively being developed and is nowhere near ready to run. Personally I prefer long-lived feature branches; this is a risk with no payoff. As for experimentation, I like percentage rollouts with segregated control/treatment group metrics. I agree that trivial on/off flags should be replaced by code deployments where possible (at my day job, code deployments happen to be a lot slower).
  - hinkley 3y ago
    Companies have an expectation that most of the code they paid for will eventually ship. Pretending like code might not ever be finished is an odd choice. The exception is experimental work. Obviously experiments aren't being merged into master, so they don't need to be included in the arithmetic for any processes that eventually involve master.
  - eddd-ddde 3y ago
    Personally, long-lived branches feel like disaster waiting to materialize since by definition they do not integrate with upstream. Yeah, you might be really responsible and rebase and test your feature frequently, but what about other people? They are more likely to break the integration with your code since they can't even see it.
- 3y ago
perlgeek 3y ago
Feature flags give you orthogonality. When you have an `if False:` or equivalent in code, you have to deploy to enable a new feature, so if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature. Another feature of feature flags is that a non-coder can toggle them. If you need neither of these, sure, go ahead with commits and deployments instead.
- hinkley 3y ago
  When you're trying to triage a production problem or explain what happened between Saturday and today, orthogonality isn't a feature, it's a challenge to be overcome.
- JimDabell 3y ago
  > if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature. This is a symptom of long release cycles. The more work you “save up” to deploy all at once, the more difficult deployments become. Everything becomes much easier when you minimise unreleased work.
- vichle 3y ago
  > Another feature of feature flags is that a non-coder can toggle them. This is not a feature, and it will get you called in during the weekend at some point.
smokel 3y ago
I have recently seen someone wrap a single function in about 10 classes of Java to turn it into a standalone application, slap on some Docker Compose magic, add some build scripts, and then continue to look proud at the feature being wholly configurable at deployment time. Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the part of the process where most effort was saved. Note that all this was used for a system that would be deployed only at one customer, and the feature would always be on. Yes, a boolean flag would have been a better solution for this. My cynicism is probably not the best approach to change the world for the better, so any hint on how to teach younger colleagues to stop snacking micro service candy is much appreciated.
- digging 3y ago
  > My cynicism is probably not the best approach to change the world for the better, so any hint on how to teach younger colleagues to stop snacking micro service candy is much appreciated. I could be way off base here, but I'd bet a lot of junior devs are just trying to stand out. They want respect, raises, promotions, and new job offers. Invisible good solutions don't bring those.
  - catchnear4321 3y ago
    junior devs are junior devs in part because they do not yet know that there is no guarantee that anyone else has been keeping track of their victories come review time. solutions are only invisible due to a lack of sufficient documentation and communication. boring solutions are part of the job. they can be fun if the goal is shifted from implementing to automating. none of this to say you are wrong, but a junior dev that succeeds at such peacocking efforts is signaling the poor health of their surrounding team as much as their eagerness to impress. where was the manager or senior dev to say “great! have you considered a boolean?”
    - blq10 3y ago
      Nobody cares about invisible good solutions, because invisible solutions are simple - and developers are to one degree or another, rated on the degree of complexity they can deal with and how smart they can be. It is common here to understand the cleverness behind just doing the dumb thing, but it is not common in many places.
huac 3y ago
I wanted a very simple feature flagging solution, so I wrote my own with Google Sheets as the backend: https://github.com/stillmatic/flagsheet https://github.com/stillmatic/flagsheet It's slightly more complex than a code push but still very easy to use.
- calderwoodra 3y ago
  Really nice and simple feature rich solution. Thanks for sharing!
javajosh 3y ago
Easy deployments are great, and all too rare. But feature flags are good too. It's actually a trade-off between determinism and control. No flags make the blob deterministic; flags give you control over the blob very late in its runtime. In the limit your feature flags might be set by an admin endpoint (which is very common, actually). But the OP's real issue is git, and not wanting to work on a branch. I actually think that's a valid goal! I've noticed that many teams have simply moved "one level up" with their git usage. The repo is the new directory; the push is the new commit; etc. You can practice and get fast with it, but it's still too many moving parts and plenty of things to go wrong. I actually think that feature flags are too coarse anyway to block off incomplete code; you should probably use conditionals for incomplete code, even if you also have robust feature flag support. The reason is that you never, ever want to dynamically turn incomplete code on, so coupling the flag to a deployment makes perfect sense.
daigoba66 3y ago
As an aside, those “feature flags as a service” tools have some neat features, but are just way too expensive for what most apps probably need: simple binary flags that can change at runtime. Example: just use a database table. Query and cache in memory for 60s. If you want, build a simple internal web page or tool to toggle flags. This works, and scales (from experience). Do many apps really need A/B testing or segmented rollout? Maybe, but probably not.
- samtho 3y ago
  This technique has been used since Wordpress days and earlier. At it’s core, it’s a “meta” table with a string identifier and some value. It’s a pattern I still use today in my own web apps.
- mvdtnz 3y ago
  "Just use a database table" solves for the simplest cases - a single monolithic app. It works incredibly well in this case. It falls apart when I need to turn features on across multiple apps together.
  - ehnto 3y ago
    I think OPs point was a simple app doesn't need this whole suite of tools, but you pay for them anyway, which makes it a bad cost/value proposition for simple apps. So yes a more complex situation could benefit from all this extra tooling and complexity but otherwise it is dead weight/cost.
  - mrweasel 3y ago
    Then you use etcd or consul. I would claim that release a new feature by enabling additional code path across multiple apps at once is a bit of an anti-pattern. It seems rather dangerous and error prone. In that case I'd actually release it in reverse order, so to speak. Release the apps that use the services of others first and have it check is this service is available/functional and if not, skip calling it. The release the feature to the next service down the stack. Then you can always rollback the last service and be confident that the callers still work. It's way more work and I can see why for certain types of application isn't not really worth the trouble.
totallywrong 3y ago
TIL there's an industry around feature flags. I can see how it can be useful, but imo if you're doing the modern approach with k8s, gitops, etc. deployments are too easy to bother with this.
- vrosas 3y ago
  Even if deployments took 2 seconds you’ll still find a frontend engineer who wants to “iterate faster” and an ear attached to the purse strings who will listen.
- matsemann 3y ago
  Often it's not about only turning something on/off (a simple bool). But doing a slow ramp up, only enable for certain cohorts to test etc. Things still doable in code, of course, but it's nice with a dashboard to control it. You quickly end up with a NIH thing. You make a boolean toggle in the db. Then make an internal dashboard. Then some more advanced toggles. And suddenly you've spent a few weeks of coding time to essentially save yourself from spinning up an Unleash container.
8ig8 3y ago
Hello, PHP
ketchupdebugger 3y ago
>Much of the time I’m using flags so I can commit unfinished code because I don’t want to create a separate branch and have one source of truth This seems like an incident in the making. If each dev on a team commits unfinished code into prod behind flags, the whole project is going to be littered with flags and unfinished code. Some intern is going to delete a flag or an if check and then everything is going to break.
- riffraff 3y ago
  Why would an intern remove a flag? The flag is not special, it's code like everything else, with tests, ownership, documentation etc.. The idea of the "unfinished code behind a flag" is to be able to work on trunk instead of a long lived branch, increasing the pace of development and reducing integration costs. This works quite well in my experience, and definitely better than "let's keep a huge branch in sync with the main one for 3 months while we finish". The problem IME is the opposite: flags do not get removed fast enough, littering the code past their utility.
- hsn915 3y ago
  There's nothing special about flags that makes them more likely for an intern to delete by mistake. It's just code. If the team is so bad that an intern can mess things up, they will, and the mess will have nothing to do with feature flags.
  - sverhagen 3y ago
    Aren't there common patterns of good "just code" and bad "just code"? I've been told for a long time that global variables are a bad pattern. Maybe feature flags are a bad pattern too. One concern about feature flags is testing, and the added permutations of testing needed to include all the feature flags in testing. You tested with flag A on and off, you tested with flag B on and off, but did you ever test with them both on and both off? Without feature flags, a big change that could have been represented by a feature flag would hopefully have to make its way past some quality gates. With feature flags, the exact permutation that you're going to cause later today by flipping on some feature flags may well not have been tested. Not that forgetting to test is something you can't protect yourself against with tools and processes, but testing all the permutations may be expensive. You may not have to test all the permutations, if you can predict which permutations are relevant for your flipping feature flags later today. But a lot of organizations have poor discipline in cleaning up old feature flags, so it may not be so predictable. Maybe that's not a feature flag problem but an organizational problem, but the feature flags are gonna get blamed at some point, nonetheless.
once_inc 3y ago
> I did an inventory of my recent feature flags and realized that about 80% of them aren’t there to roll things out to specific populations, or do any sort of A/B testing, but to hide unfinished code. Which means that for 20% you actually do need the feature flags. So you can't do away with the product, and as the author and many of the respondents here have mentioned, there is also a number of benefits to using feature flags. In the trivial case of a feature for a single user still in development; yeah, a boolean is enough. But that is not the intended use case for feature flags, nor is it what the libraries promote you to do.
vinay_ys 3y ago
Pushing config to production is also deployment. Pushing config update to a dynamic config serving service from which all other services pull the config is also deployment. You need safety guardrails for any deployment of code or config – a clearly defined unit of deployment that is version controlled, changes going through a repeatable automated flow in which lints/tests can be added, changes can be rolled out atomically all at once, or rolled out with careful control gradually, can be rolled back to a known safe version instantly when needed etc. All of this applies to config and code (binary). Within code, having feature flags, dynamic config variables, experiments that can be ramped up slowly and ramped down automatically upon regression detection etc – these are mechanisms to make your life easier when you have unknown-unknowns manifest to bite you. And they always do in any non-trivial real-world large-scale distributed architecture application that needs to evolve continuously. And these unknown-unknowns are not just code bugs, they are mix shifts in usage patterns, data etc causing unanticipated behaviors too. They are old stale stored state interacting with new code that you didn't test for. They are changes in code outside of your control – like remote services or OS/language libraries – that you assume to be mostly stable and don't exhaustively test every single time – that you don't even know when they changed - that change behavior without changing interface etc. These are real-world software systems problems. If your software is built like an appliance – frozen in time and doesn't change ever after, is stateless, doesn't interact with outside world etc – then you don't need any of those guardrails and you can keep your code simple.
Palmik 3y ago
At Google, there are no branches, so typically development indeed happens under some experiment flag. Binary, configs and experiment flags are on separate deployment schedules, and often you need to coordinate your release across different systems (with their own binary, config and experiment flag deployment schedules). This requires you to think about backward & forward compatibility of all your changes.
bullen 3y ago
With my SaaS/PaaS you hot-deploy async. to all global nodes with complete (type last code character to ready) turnaround of ~1-2 seconds on live (just enough time to switch to your client/browser): http://github.com/tinspin/rupy http://github.com/tinspin/rupy It also allows for session persistence, so you can test the dev/bug feature without logging in again. And yes code becomes more manageable if you work like this because it encourages modularity.
wrsh07 3y ago
> Much of the time I’m using flags so I can commit unfinished code There are two interpretations of this, and I find one of them horrifying 1. I'm working on a feature, I need to update multiple components, so I make small changes for each component and feature flag those (this is fine! Totally normal) 2. I commit code that doesn't work because I'm unwilling to use or understand git The second of these scares the crap out of me. Why? Doesn't it feel like a bit of a footgun to have broken code deployed? However, I like most of the other comments of the essay - deploys should be easy! That makes rollbacks safe and easy. Using bool flags is fine too - imo feature flagging systems are best at de risking expensive deploys. If you have an expensive deploy (thinking at least 20m to deploy prod) and you have staging which mirrors prod (probably also 15-20m to deploy staging), then any commit you want to deploy to fix or revert a change in prod is going to cost you a lot of time or hassle. Feature flags give you a mechanism to instantly rollback. But it sounds like you already have that - it's called your normal deploy.
- WorldMaker 3y ago
  "Broken code" "deployed" inside an `if (false) {}` (or even an `#if FALSE #endif` depending on your language of choice) isn't that much more of a footgun than a git branch hundreds of commits behind and full of merge issues. It's something of a "different strokes" preference and it's something of a question of how fast the main branch is moving and how hard to keep up with it. Most language compilers and JITs eliminate the unreachable code within a constant false branch sooner or later and the code isn't even technically "deployed" even though it exists in source control for that deployment.
  - wrsh07 3y ago
    I would strongly recommend learning git basics! My branches are never more than a day behind main (rebase does wonders!) Code inside an if false block might as well be commented, and commented code is the fastest to bitrot - I would strongly recommend against!
    - WorldMaker 3y ago
geniium 3y ago
Doesn’t that feels like click bait? « When developer are clever, codes become better »
SergeAx 3y ago
Is it just me, or there's no actual difference in simplicity between global constant myExperiment=false and well abstracted A/B experiment library?

189 points by recroad 3y ago

I'm confused. This is standard practice for CI/CD, but nobody is mentioning it. Is CI/CD that obscure?

dmoy 3y ago

> Is this bad because I now have to do a deployment to enable the feature? ... I would argue as long as your deployments are easy, this is the better way to do things because it reduces the complexity of integrating with a third-party tool. Shoot, even when it's all first party tooling, I prefer a release-flags-and-binary as an atomic unit. If the flags and binary are going out as a single push, it simplifies a lot of things: * reproducible state for given time with only one thing to track (running version), instead of two or more things (binary version, flag configs version) * corollary: rollbacks are much more simple, because it's just one thing to rollback - the code+config+everything as an atomic unit, not "did we need to roll back the flags, or the binary, or both?" (Corollary to corollary: removes the awful problem of "oh shit the code and/or flags/configs wasn't backwards/forwards compatible" where you shoot yourself in the foot while doing a rollback * removals and cleanups are easier: can remove flags, config bits, and code all at once, instead of having to do a careful dance of "set code behavior to default-on and remove dependency on flag, the wait for full deployment, then make sure everyone is okay with not rolling back after a given point, then remove flags and config logic" * depending on your tooling, diffs vs prod during code review get cleaner

ljm 3y ago
The problem is when your deployment depends on external state like a database. Code rollbacks are trivial, rolling back state (if you even can) is not.
- mcny 3y ago
  I always say this - there is no such thing as a rollback. Anyone who pretends you can simply "roll back" a stateful application is out of their mind. If you can roll back and forward events, good for you but for most of us rollbacks are actually a new build, deployment, and a new set of tests. Every single time.
  - perrygeo 3y ago
    The system can only move forwards. Just like a `git revert`, it's a new commit to an immutable history. This is why I've never used or understood the value of the "downgrade" feature in some database migration tools. If you need to revert, make a _new_ migration that fixes the problem. Your tooling/code/logs should reflect the true history of the system without cooking the books.
    - OJFord 3y ago
      In what sense is 'downgrade' not just that prepared ahead of time? Or do you just not like the name? Of course usually it would be followed by something that fixes it 'properly' - applying the original 'upgrade' again but with some correction, but typically that'll take longer than reverting to the previous version, and given it's gone wrong you'll probably want to take longer with it, test more, restore confidence.
      - perrygeo 3y ago
        downgrade implies that such a thing is possible. In a stateful system though, you cannot take back that which you have committed. It's quite literally impossible by definition.
        OJFord 3y ago
        No it isn't? Forward migration: add date column, set = date(day, month, year) for existing records, drop is valid date constraint, drop day/month/year columns Down migration: add day/month/year columns, add is valid date constraint, set day/month/year = appropriate datepart of date, drop date column for trivial example.
- chaxor 3y ago
  If a git commit hash was directly tied to a data hash at that state (IPFS), that would be trivial.
  - tbrownaw 3y ago
    Or have the SAN make a snapshot of your db before the deploy, no fancy solution-looking-for-a-problem tech needed. Completely reverting all the state to some point in the past is easy and have been a solved problem for quite a while. The actual problem is undoing states changes related to the code change, while not losing state changes related to normal system activity. If I add some new fields to how my web forum records posts, and then find out that it's eating every tenth post and need to revert, it'd be good to not lose and posts made in the meantime.
    - chaxor 3y ago
      Good point, this system is much more useful for scientists and ML researchers, etc. wherein the code may generate datasets over several weeks or months of compute time, but produce some output that should be directly tied to the code. You're right though. Systems in the social media space dealing with users appending to a database typically don't have this requirement to be easily reproducible, so the local snapshots are the right choice.
  - baq 3y ago
    And useless. Unless customers don’t care about their data.
    - chaxor 3y ago
      I don't understand your statement. If a git clone of a specific commit also pulls the data that is generated from that commit hash via IPFS simultaneously, then you can always have the data at whatever state you choose. How might this have a negative effect on conserving the data? Do you mean that there are additional structures required to manage keys if the data is encrypted?
  - rjmunro 3y ago
- dmoy 3y ago
  Sure, that is still a problem. But that problem exists whether or not your flags/configs are pushed atomically with your code/binary changes. Having flags push separately adds another problem on top of data corruption.
- mrweasel 3y ago
  One of the very best development teams I worked with had an interesting take, they always did database migrations first. Any new state that was to be added to the system could only be done so by first adding the new database fields or tables. This ensure that version 1 of the code would work with version 1 and 2 of the database. They would then roll out version 2 of the code, but have the new features hidden behind a feature flags (in the database), ensuring that version 2 could run, without using the new database schema. Once they where confident that everything was still running on version 2 of the code and database they'd enable the new feature. Later the feature flag could be migrated from the database to a properties file, saving the database lookup. I wouldn't necessarily call this approach simple, but it was incredibly safe and rollbacks was always a none event.
  - steffres 3y ago
    That sounds reasonable. But what about the case where the DB migration of version 2 would be incompatbile with code version 1, e.g. a column was dropped?
    - jaimebuelta 3y ago
      You NEVER do that in one go, you need to split it in several deployments. Dropping a column is relatively straightforward, in two steps. First deploy a version of the code that doesn’t use the column, then release the migration dropping the column. The typical example is the renaming of a column, which needs to be done in several steps: 1. Create the new column, copying the data from the old column (DB migration) both columns exist, but only the old one is used 2. Deploy new code that works with both columns, reading from old and writing new and old 3. Deploy data migration (DB migration) that ensures old and new columns has the same values (to ensure data consistency). At this point, there are no “old column only” writes by the code deployed in previous step 4. Deploy new code using only new column. Old column is deprecated 5. Delete old column At any given point, code versions N (current) and N-1 (previous) are compatible with the DB. Any change on the DB is done in advance in a backwards compatible way.
      - 3y ago
gonzan 3y ago
I agree with all this and it’s how I do it for web development but the time Apple/Google take to approve apps makes doing this for mobile apps quite risky since it’s hard to rollback. I guess it goes to the point of the author that hard deployments on mobile makes mobile development harder

teeray 3y ago

Anyone have advice for teams without control over when their software is deployed by the people actually running it? Say you have a product released to a third party and they may or may not take releases and may be arbitrarily delayed in deploying those they choose to take.

GrinningFool 3y ago
You're in a for a long tail of pain if you have significant uptake. To avoid this, you will need have a clear policy on what you're willing to support (eg versions up to 1 year old, or "major version - 1", etc.), and stick to it. Otherwise, people will expect you to support all of it, forever.
- hinkley 3y ago
  Bug fix releases never get deployed. If you want people to keep up to date with your releases, you either have to provide an automatic update facility that they have to manually turn off (ugh, too hard), or you have to dangle desirable features as carrots in front of them, so that upgrading is exciting instead of a social obligation.
  - jcrben 3y ago
    My experience is kinda the opposite. Big companies get into certain workflows and don't want to change, altho they are happy to take bugfixes. This is also where I'm at with much of the software I rely upon, some of which I pull at my own whim.
GauntletWizard 3y ago
Give yourself as much leverage as you can get over their deployment cycles. Use both carrot and stick - Include features they can't live without with the hardest of deployment maintenance burdens, and make sure you have tight adherence to your software lifecycle - Every question about 1.0 should be answered with "We don't know, 1.0 is out of support" from the second that it's EOL date passes. Include the currently running version in every logfile, if not every log statement. Make it clear to you where they are and run your own run-behind that upgrades only after they do. This will be painful, but it will save you overall time.

bsima 3y ago

> I did an inventory of my recent feature flags and realized that about 80% of them aren’t there to roll things out to specific populations, or do any sort of A/B testing, but to hide unfinished code. Maybe it’s just me but having unfinished, dead, or scratch code in a production codebase really annoys me. Either finish your work or delete the unneeded code. More than a few times I’ve sunk time out of my day into investigating some code path only to realize it’s completely unused.

deleted 3y ago
[deleted]
erik_seaberg 3y ago
“Unfinished” can mean abandoned, but it can also mean half-baked code that has been pushed to master even though it’s still actively being developed and is nowhere near ready to run. Personally I prefer long-lived feature branches; this is a risk with no payoff. As for experimentation, I like percentage rollouts with segregated control/treatment group metrics. I agree that trivial on/off flags should be replaced by code deployments where possible (at my day job, code deployments happen to be a lot slower).
- hinkley 3y ago
  Companies have an expectation that most of the code they paid for will eventually ship. Pretending like code might not ever be finished is an odd choice. The exception is experimental work. Obviously experiments aren't being merged into master, so they don't need to be included in the arithmetic for any processes that eventually involve master.
- eddd-ddde 3y ago
  Personally, long-lived branches feel like disaster waiting to materialize since by definition they do not integrate with upstream. Yeah, you might be really responsible and rebase and test your feature frequently, but what about other people? They are more likely to break the integration with your code since they can't even see it.
3y ago

perlgeek 3y ago

Feature flags give you orthogonality. When you have an `if False:` or equivalent in code, you have to deploy to enable a new feature, so if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature. Another feature of feature flags is that a non-coder can toggle them. If you need neither of these, sure, go ahead with commits and deployments instead.

hinkley 3y ago
When you're trying to triage a production problem or explain what happened between Saturday and today, orthogonality isn't a feature, it's a challenge to be overcome.
JimDabell 3y ago
> if the current state in the test environment isn't good, you either have to wait for it to be fixed, or to roll back all possibly bad new commits before you can enable the feature. This is a symptom of long release cycles. The more work you “save up” to deploy all at once, the more difficult deployments become. Everything becomes much easier when you minimise unreleased work.
vichle 3y ago
> Another feature of feature flags is that a non-coder can toggle them. This is not a feature, and it will get you called in during the weekend at some point.

smokel 3y ago

I have recently seen someone wrap a single function in about 10 classes of Java to turn it into a standalone application, slap on some Docker Compose magic, add some build scripts, and then continue to look proud at the feature being wholly configurable at deployment time. Of course, the deployment team would have to be informed about this change, so some documentation was required as well, but that was exactly the part of the process where most effort was saved. Note that all this was used for a system that would be deployed only at one customer, and the feature would always be on. Yes, a boolean flag would have been a better solution for this. My cynicism is probably not the best approach to change the world for the better, so any hint on how to teach younger colleagues to stop snacking micro service candy is much appreciated.

digging 3y ago
> My cynicism is probably not the best approach to change the world for the better, so any hint on how to teach younger colleagues to stop snacking micro service candy is much appreciated. I could be way off base here, but I'd bet a lot of junior devs are just trying to stand out. They want respect, raises, promotions, and new job offers. Invisible good solutions don't bring those.
- catchnear4321 3y ago
  junior devs are junior devs in part because they do not yet know that there is no guarantee that anyone else has been keeping track of their victories come review time. solutions are only invisible due to a lack of sufficient documentation and communication. boring solutions are part of the job. they can be fun if the goal is shifted from implementing to automating. none of this to say you are wrong, but a junior dev that succeeds at such peacocking efforts is signaling the poor health of their surrounding team as much as their eagerness to impress. where was the manager or senior dev to say “great! have you considered a boolean?”
  - blq10 3y ago
    Nobody cares about invisible good solutions, because invisible solutions are simple - and developers are to one degree or another, rated on the degree of complexity they can deal with and how smart they can be. It is common here to understand the cleverness behind just doing the dumb thing, but it is not common in many places.

huac 3y ago

I wanted a very simple feature flagging solution, so I wrote my own with Google Sheets as the backend: https://github.com/stillmatic/flagsheet https://github.com/stillmatic/flagsheet It's slightly more complex than a code push but still very easy to use.

calderwoodra 3y ago
Really nice and simple feature rich solution. Thanks for sharing!

javajosh 3y ago

Easy deployments are great, and all too rare. But feature flags are good too. It's actually a trade-off between determinism and control. No flags make the blob deterministic; flags give you control over the blob very late in its runtime. In the limit your feature flags might be set by an admin endpoint (which is very common, actually). But the OP's real issue is git, and not wanting to work on a branch. I actually think that's a valid goal! I've noticed that many teams have simply moved "one level up" with their git usage. The repo is the new directory; the push is the new commit; etc. You can practice and get fast with it, but it's still too many moving parts and plenty of things to go wrong. I actually think that feature flags are too coarse anyway to block off incomplete code; you should probably use conditionals for incomplete code, even if you also have robust feature flag support. The reason is that you never, ever want to dynamically turn incomplete code on, so coupling the flag to a deployment makes perfect sense.

daigoba66 3y ago

As an aside, those “feature flags as a service” tools have some neat features, but are just way too expensive for what most apps probably need: simple binary flags that can change at runtime. Example: just use a database table. Query and cache in memory for 60s. If you want, build a simple internal web page or tool to toggle flags. This works, and scales (from experience). Do many apps really need A/B testing or segmented rollout? Maybe, but probably not.

samtho 3y ago
This technique has been used since Wordpress days and earlier. At it’s core, it’s a “meta” table with a string identifier and some value. It’s a pattern I still use today in my own web apps.
mvdtnz 3y ago
"Just use a database table" solves for the simplest cases - a single monolithic app. It works incredibly well in this case. It falls apart when I need to turn features on across multiple apps together.
- ehnto 3y ago
  I think OPs point was a simple app doesn't need this whole suite of tools, but you pay for them anyway, which makes it a bad cost/value proposition for simple apps. So yes a more complex situation could benefit from all this extra tooling and complexity but otherwise it is dead weight/cost.
- mrweasel 3y ago
  Then you use etcd or consul. I would claim that release a new feature by enabling additional code path across multiple apps at once is a bit of an anti-pattern. It seems rather dangerous and error prone. In that case I'd actually release it in reverse order, so to speak. Release the apps that use the services of others first and have it check is this service is available/functional and if not, skip calling it. The release the feature to the next service down the stack. Then you can always rollback the last service and be confident that the callers still work. It's way more work and I can see why for certain types of application isn't not really worth the trouble.

totallywrong 3y ago

TIL there's an industry around feature flags. I can see how it can be useful, but imo if you're doing the modern approach with k8s, gitops, etc. deployments are too easy to bother with this.

vrosas 3y ago
Even if deployments took 2 seconds you’ll still find a frontend engineer who wants to “iterate faster” and an ear attached to the purse strings who will listen.
matsemann 3y ago
Often it's not about only turning something on/off (a simple bool). But doing a slow ramp up, only enable for certain cohorts to test etc. Things still doable in code, of course, but it's nice with a dashboard to control it. You quickly end up with a NIH thing. You make a boolean toggle in the db. Then make an internal dashboard. Then some more advanced toggles. And suddenly you've spent a few weeks of coding time to essentially save yourself from spinning up an Unleash container.

8ig8 3y ago

Hello, PHP

ketchupdebugger 3y ago

>Much of the time I’m using flags so I can commit unfinished code because I don’t want to create a separate branch and have one source of truth This seems like an incident in the making. If each dev on a team commits unfinished code into prod behind flags, the whole project is going to be littered with flags and unfinished code. Some intern is going to delete a flag or an if check and then everything is going to break.

riffraff 3y ago
Why would an intern remove a flag? The flag is not special, it's code like everything else, with tests, ownership, documentation etc.. The idea of the "unfinished code behind a flag" is to be able to work on trunk instead of a long lived branch, increasing the pace of development and reducing integration costs. This works quite well in my experience, and definitely better than "let's keep a huge branch in sync with the main one for 3 months while we finish". The problem IME is the opposite: flags do not get removed fast enough, littering the code past their utility.
hsn915 3y ago
There's nothing special about flags that makes them more likely for an intern to delete by mistake. It's just code. If the team is so bad that an intern can mess things up, they will, and the mess will have nothing to do with feature flags.
- sverhagen 3y ago
  Aren't there common patterns of good "just code" and bad "just code"? I've been told for a long time that global variables are a bad pattern. Maybe feature flags are a bad pattern too. One concern about feature flags is testing, and the added permutations of testing needed to include all the feature flags in testing. You tested with flag A on and off, you tested with flag B on and off, but did you ever test with them both on and both off? Without feature flags, a big change that could have been represented by a feature flag would hopefully have to make its way past some quality gates. With feature flags, the exact permutation that you're going to cause later today by flipping on some feature flags may well not have been tested. Not that forgetting to test is something you can't protect yourself against with tools and processes, but testing all the permutations may be expensive. You may not have to test all the permutations, if you can predict which permutations are relevant for your flipping feature flags later today. But a lot of organizations have poor discipline in cleaning up old feature flags, so it may not be so predictable. Maybe that's not a feature flag problem but an organizational problem, but the feature flags are gonna get blamed at some point, nonetheless.

once_inc 3y ago

> I did an inventory of my recent feature flags and realized that about 80% of them aren’t there to roll things out to specific populations, or do any sort of A/B testing, but to hide unfinished code. Which means that for 20% you actually do need the feature flags. So you can't do away with the product, and as the author and many of the respondents here have mentioned, there is also a number of benefits to using feature flags. In the trivial case of a feature for a single user still in development; yeah, a boolean is enough. But that is not the intended use case for feature flags, nor is it what the libraries promote you to do.

vinay_ys 3y ago

Pushing config to production is also deployment. Pushing config update to a dynamic config serving service from which all other services pull the config is also deployment. You need safety guardrails for any deployment of code or config – a clearly defined unit of deployment that is version controlled, changes going through a repeatable automated flow in which lints/tests can be added, changes can be rolled out atomically all at once, or rolled out with careful control gradually, can be rolled back to a known safe version instantly when needed etc. All of this applies to config and code (binary). Within code, having feature flags, dynamic config variables, experiments that can be ramped up slowly and ramped down automatically upon regression detection etc – these are mechanisms to make your life easier when you have unknown-unknowns manifest to bite you. And they always do in any non-trivial real-world large-scale distributed architecture application that needs to evolve continuously. And these unknown-unknowns are not just code bugs, they are mix shifts in usage patterns, data etc causing unanticipated behaviors too. They are old stale stored state interacting with new code that you didn't test for. They are changes in code outside of your control – like remote services or OS/language libraries – that you assume to be mostly stable and don't exhaustively test every single time – that you don't even know when they changed - that change behavior without changing interface etc. These are real-world software systems problems. If your software is built like an appliance – frozen in time and doesn't change ever after, is stateless, doesn't interact with outside world etc – then you don't need any of those guardrails and you can keep your code simple.

Palmik 3y ago

At Google, there are no branches, so typically development indeed happens under some experiment flag. Binary, configs and experiment flags are on separate deployment schedules, and often you need to coordinate your release across different systems (with their own binary, config and experiment flag deployment schedules). This requires you to think about backward & forward compatibility of all your changes.

bullen 3y ago

With my SaaS/PaaS you hot-deploy async. to all global nodes with complete (type last code character to ready) turnaround of ~1-2 seconds on live (just enough time to switch to your client/browser): http://github.com/tinspin/rupy http://github.com/tinspin/rupy It also allows for session persistence, so you can test the dev/bug feature without logging in again. And yes code becomes more manageable if you work like this because it encourages modularity.

wrsh07 3y ago

> Much of the time I’m using flags so I can commit unfinished code There are two interpretations of this, and I find one of them horrifying 1. I'm working on a feature, I need to update multiple components, so I make small changes for each component and feature flag those (this is fine! Totally normal) 2. I commit code that doesn't work because I'm unwilling to use or understand git The second of these scares the crap out of me. Why? Doesn't it feel like a bit of a footgun to have broken code deployed? However, I like most of the other comments of the essay - deploys should be easy! That makes rollbacks safe and easy. Using bool flags is fine too - imo feature flagging systems are best at de risking expensive deploys. If you have an expensive deploy (thinking at least 20m to deploy prod) and you have staging which mirrors prod (probably also 15-20m to deploy staging), then any commit you want to deploy to fix or revert a change in prod is going to cost you a lot of time or hassle. Feature flags give you a mechanism to instantly rollback. But it sounds like you already have that - it's called your normal deploy.

WorldMaker 3y ago
"Broken code" "deployed" inside an `if (false) {}` (or even an `#if FALSE #endif` depending on your language of choice) isn't that much more of a footgun than a git branch hundreds of commits behind and full of merge issues. It's something of a "different strokes" preference and it's something of a question of how fast the main branch is moving and how hard to keep up with it. Most language compilers and JITs eliminate the unreachable code within a constant false branch sooner or later and the code isn't even technically "deployed" even though it exists in source control for that deployment.
- wrsh07 3y ago
  I would strongly recommend learning git basics! My branches are never more than a day behind main (rebase does wonders!) Code inside an if false block might as well be commented, and commented code is the fastest to bitrot - I would strongly recommend against!
  - WorldMaker 3y ago

geniium 3y ago

Doesn’t that feels like click bait? « When developer are clever, codes become better »

SergeAx 3y ago

Is it just me, or there's no actual difference in simplicity between global constant myExperiment=false and well abstracted A/B experiment library?