7 ms·
The single biggest value add of feature flags is that they de-risk deployment. They make it less frightening and difficult to turn features on and off, which me
by simonw 2y ago
The single biggest value add of feature flags is that they de-risk deployment. They make it less frightening and difficult to turn features on and off, which means you'll do it more often. This means you can build more confidently and learn faster from what you build. That's worth a lot.
I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change and using an (often expensive) feature flags as a service platform: roll your own simple system.
A relational database lookup against primary keys in a table with a dozen records is effectively free. Heck, load the entire collection at the start of each request - through a short lived cache if your profiling says that would help.
Once you start getting more complicated (flags enabled for specific users etc) you should consider build-vs-buy more seriously, but for the most basic version you really can have no-deploy-changes at minimal cost with minimal effort.
There are probably good open source libraries you can use here too, though I haven't gone looking for any in the last five years.
- tdumitrescu 2y agoSeriously. This is one of those cases where rolling your own really does make sense. Flags in a DB table, flags in a json file, all super simple to build and maintain, and 100x faster and more reliable than making the critical paths of your application's request cycle depend on an external provider.
- joshmanders 2y agoYou know what I would find worse than telling my customers that they can't access the application they paid for and works because I farmed my auth out to a 3rd party that is having an outage? Telling them that my auth provider isn't out, but the thing I use to show them a blue button vs a red button is. Oof.
- deleted 2y ago[deleted]
- gboss 2y agoHas this actually been a problem? We’ve been using launch darkly for years and if they do have an outage (which is really really rare) the flag will be set to the default value. It’s also very very cheap, maybe $500 a month.
- joshmanders 2y ago$500 a month and it has had many major outages in 2024 alone. lol https://status.launchdarkly.com/uptime?page=5 https://status.launchdarkly.com/uptime?page=5
- deployhq 2y agoWhat about ConfigCat?
- joshmanders 2y agoI don't know, why don't you do the research and let us know?
- helen_devunus 2y ago[dead]
- PaulHoule 2y agoIn years of trying to sell things I've found that one of the best selling points to management is "susceptible to vendor lock-in", "you don't own your customer database", etc. I have no idea why that is.
- dasil003 2y agoI'm confused. Are you saying this ironically or have you literally pitched management with the risks of using your product?
- PaulHoule 2y agoI developed an open source "user management system" circa 2001 which I used on several sites, including one that had 400,000+ users, a famous preprint archive and the web site of a county-level green party. It was patterned on what sites like Yahoo and Amazon had at the time, did email verification and numerous things that were a hassle to implement, had great screens for the administrators, all of that. I couldn't get anybody else to adopt this software despite putting a lot of work into making it easy to pick up and install. 10 years later competitors popped up like mushrooms and were adopted quickly. The thing they all had in common was somebody else owned your user database. So yeah I feel pretty cynical.
- cyberax 2y agoThere's such a thing as being too early. My university had a great shared browser bookmark management system, even with a basic discussion support for them. In 1998. It was not super popular because people just didn't have that many links to share, eventually it fell offline and got accidentally deleted in 2001.
- twisteriffic 2y agoWe did this. Two tables. One for feature flags, with name, desc, id, enum (none, defaultToEnabled, overrideToDisabled). One for user flag overrides, with flagId, userId, enum (enabled, disabled). The combination of these two has been all we've ever needed. User segmentation, A/B testing, pilot soft launch etc are all easy.
- uutangohotel 2y agoWould you mind expanding on the usage of enums for the feature flags table? Why not use a boolean?
- twisteriffic 2y agoWe actually did use booleans, I just found it easier to explain using enums, and the code would have been simpler if we'd done it that way.
- echelon 2y ago> build-vs-buy Roll your own. Seriously. Feature flags are such an easy thing that there should be a robust and completely open source offering not tied to B2B SaaS. Until then, do it in house. My team built a five nines feature flag system that handled 200k QPS from thousands of services, active-active, local client caching, a robust predicate DSL for matching various conditions, percent rollout, control plane, ACLs, history, everything. It was super robust and took half an engineer to maintain. We ultimately got roped into the "build vs buy" / "anti-weirdware" crosshairs from above. Being tasked with migrating to LaunchDarkly caused more outages, more headache, and more engineering hours spent. We were submitting fixes to LaunchDarkly's code, fixing the various language client integrations, and writing our own Ruby batching and multiprocessing. And they charged us way more for the pleasure. Huge failure of management. I've been out of this space for some years now, but someone should "Envoy" this whole problem and be done with it. One service, optional sidecars, all the language integrations. Durable failure and recovery behavior. Solid UX. This shouldn't be something you pay for. It should be a core competency and part of your main tooling.
- rav 2y agoI don't understand what a dedicated "completely open source offering" provides or what your "five nines feature flag system" provides. If you're running on a simple system architecture, then you can sync some text files around, and if you have a more scalable distributed architecture, then you're probably already handling some kind of slowly-changing, centrally-managed system state at runtime (e.g. authentication/authorization, or in-app news updates, ...) where you can easily add another slowly-changing, centrally-managed bit of data to be synchronised. How do you measure the nines on a feature flag system, if you're not just listing the nines on your system as a whole?
- foobazgt 2y ago> If you're running on a simple system architecture, His point was that even a feature flag system in a complex environment with substantial functional and system requirements is worth building vs buying. If your needs are even simpler, then this statement is even more true! I'm having a hard time making sense out of the rest of your comment, but in larger businesses the kinds of things you're dealing with are: - low latency / staleness: You flip a flag, and you'll want to see the results "immediately", across all of the services in all of your datacenters. Think on the order of one second vs, say 60s. - scalability: Every service in your entire business will want to check many feature flags on every single request. For a naive architecture this would trivially turn into ungodly QPS. Even if you took a simple caching approach (say cache and flush on the staleness window), you could be talking hundreds of thousands of QPS across all of your services. You'll probably want some combination of pull and push. You'll also need the service to be able to opt into the specific sets of flags that it cares about. Some services will need to be more promiscuous and won't know exactly which flags they need to know in advance. - high availability: You want to use these flags everywhere, including your highest availability services. The best architecture for this is that there's not a hard dependency on a live service. - supports complex rules: Many flags will have fairly complicated rules requiring local context from the currently executing service call. Something like: "If this customer's preferred language code is ja-JP, and they're using one of the following devices (Samsung Android blah, iPhone blargh), and they're running versions 1.1-1.4 of our app, then disable this feature". You don't want to duplicate this logic in every individual service, and you don't want to make an outgoing service call (remember, H/A), so you'll be shipping these rules down to the microservices, and you'll need a rules engine that they can execute locally. - supports per-customer overrides: You'll often want to manually flip flags for specific customers regardless of the rules you have in place. These exclusion lists can get "large" when your customer base is very large, e.g. thousands of manual overrides for every single flag. - access controls: You'll want to dictate who can modify these flags. For example, some eng teams will want to allow their PMs to flip certain flags, while others will want certain flags hands off. - auditing: When something goes wrong, you'll want to know who changed which flags and why. - tracking/reporting: You'll want to see which feature flags are being actively used so you can help teams track down "dead" feature flags. This list isn't exhaustive (just what I could remember off the top of my head), but you can start to see why they're an endeavor in and of themselves and why products like LaunchDarkly exist.
- Kwpolska 2y ago> using an (often expensive) feature flags as a service platform I have no idea why anyone would actually do that in real life. Feature flags are something so trivial that you can implement them from scratch in a few hours, tops — and that includes some management UI.
- hirsin 2y agoYou'll still be building management UI over their system (it doesn't understand or validate actor types, tenants, etc, so you have to do that.). But at high throughput, you might want something with dedicated professional love. Ten thousand feature flags, being checked at around 2 (or 200) million RPS from multiple deployments... I don't want to be the team with that as their side project. And once you're talking a team of three to six engineers to build all this out, maybe it makes sense to just buy something for half a million a year. Assuming it can actually fit your model.
- Spivak 2y agoBut it's not a side-project in most implementations it's part of the app itself.
- jasonkarns 2y agoSide project means it’s not that team’s primary focus.
- fizx 2y agoThe scale is easy in practice, cause you outsource to a CDN. But everything takes time and has opportunity cost.
- hirsin 2y agoMaybe we've worked with different FF systems, but anything that involves a call more expensive than an RPC would be lethal to request latency. Calling out to a CDN forty five times per inbound request would be... Infeasible
- elliot07 2y agoI agree with a lot of this, except for the part about de-risking deployments. That should not be a reason why to adopt a feature flag platform - that is a symptom of a bad deployment pipeline that should be fixed which is a whole other story.
- duke_sam 2y agoThe risk of deployments isn’t entirely technical. Depending on your business and customer base it might be necessary for some groups to have access to the feature earlier or later than others.
- tomnipotent 2y ago> That should not be a reason why to adopt a feature flag platform It's one of the two big reasons. First is the ability to rollout features gradually and separate deployments from feature release, and second is the ability to turn new features off when something goes wrong. Even part of the motivation of A/B testing is de-risking.
- mlinhares 2y agoStrong disagree here, my whole org does not roll out changes without feature flags at all and whenever someone doesn't follow this policy they cause large scale incidents. Feature flags are actually a sign the deployment pipeline is very sane and mature, because people understand any new code comes with unexpected risks and we should prevent these risks from taking down systems.
- CBLT 2y agoI disagree that using feature flags to de-risk deployments is a symptom of bad deployment pipelines. There's several aspects of deployments that are in contention with each other: safety, deployment latency, and engineering overhead are how I'd break it down. Every deployment process is a tradeoff between these factors. What I (maybe naively) think you're advocating is writing more end-to-end tests, which moves the needle towards safety at the expense of the other factors. In particular, having end to end tests that are materially better than well-written k8s health checks (which you already have, right?) is pretty hard. They might be flakey, they might depend on a lot of specifics of the application that's subject to change, and they might just not be prioritized. In my experience, the highest value end-to-end tests are based on learned experiences of what someone already saw go wrong once. Writing comprehensive testing before the feature is even out results in many low quality tests, which is an enormous drain on productivity to write them, to maintain them, and to deal with the flakey tests. It is better, I think, to have non-comprehensive end-to-end tests that provides as much value for the lowest overhead on human resources. And the safety tradeoff we make there can be mitigated by having the feature behind a flag. My whole thesis, really, is that by using feature flags you can make better tradeoffs between these than you otherwise could.
- traverseda 2y agoWhy aren't you just using environment variables for feature flags? Have people still not bought into the whole 12 factor config things?
- jitl 2y agoWhen your app starts to get bigger and more complex, the idea of needing to restart a process to pick up any new kind of data starts to seem silly. Have seen the pattern many times: Hard-code values in code -> configure via env -> configure slow things via env and fast things via redis -> configure almost everything via a config management system I do not want to reboot every instance in a fleet of 2000 nodes just to enable a new feature for a new batch of beta testers. How do I express that in an env var anyways? What if I have 100s of flags I need to control? In other cases I need some set of nodes to behave one way, and some set of nodes to behave another way - say the nodes in us-west-2 vs the nodes in eu-central-1. Do I really want to teach my deploy system the exhaustive differences of configuration between environments? No I want my orchestration and deploy layer to be as similar as possible between regions, and push almost everything besides region & environment identification into the app layer - those two can be env vars because they basically never change for the life of the cluster.
- traverseda 2y agoHow do you do software upgrades if you don't have a good system for handling process restarts without downtime?
- jitl 2y agoSoftware updates happen once per two hours, config changes happen once per 5 minutes or faster. A few days ago I’m tuning performance parameters for a low latency stream processing system, I can iterate in 90 seconds by twiddling some config management bits for 30s in the CLI, watch the graphs for 60s, then repeat.
- zoogeny 2y agoI mean, isn't that even worse? If I have 100 servers and I'm doing rolling deploys then I'm going to be in a circumstance where some ratio of my services are in one state and some ratio are in another state. If I am reading per-request from redis (even with a server cache) I have finer-grained control. For me it is a question of "is the config valid for the life of this process" vs. "is this config something that might change while this process is alive".
- mattmanser 2y agoI've been doing this a long time and seen a few different apps use config in database. There's different levels of config you're talking about here, but general app config should generally not go in a db. No-one ever changes the bloody things and it's just an extra thing to go wrong. If it only loads on startup, it achieves nothing over a bog standard config file. If it loads every request you've just incurred a 5% overhead on every call. And it ALWAYS ends up filled with crap that doesn't work anymore. Because unlike config files, no-one clear it up. Worse still is when people haven't made it injectable and then it means unit tests rely on a real database, or it blocks getting a proper CI/CD pipeline working. I end up having to pick the damn thing out of the app. Use a config file like everyone else that's probably built into the framework you're using. To be honest, most of the time I've seen it has been when people who clearly did not know their language/framework who wrote the app. I'm not saying it's you, but that's been my honest experience of config in the db, it's generally been a serious code smell that the whole app will be bad.
- mjr00 2y agoThere's differences to what kind of configuration you'd want to have in a config file (or environment variables, or some other "system level" management tooling) versus a feature flagging system. In my experience, feature flagging is more application-level than system-level. What I mean by that is, feature flagging is for stuff like: roll this feature out to 10% of users, or to users in North America, or to users who have opted into beta features; enable this feature and report conversion metrics (aka A/B testing); enable this experimental speedup for 15 minutes so we can measure the performance increase. It's stuff that you want to change at runtime, through centralized tooling with e.g. auditing and alerting, without restarting all of your application servers. It's a bit different than config for like "what's the database host and user", stuff that you don't want to change after initialization (generally). Regarding the article though, early on your deployment pipeline should be fast enough that updating a hardcoded JSON file and redeploying is just as easy as updating a feature flag, so I agree it's not something to invest in if you're still trying to get your first 1000 users.
- marcosdumay 2y agoFor some kind of software, another call to the DB is the best way to add bog-standard functionality without adding complexity and failure modes. Granted, not for all software. And there's something to be said about a config file that you can just replace at deployment. But that's something that varies a lot from one environment to another.
- adamtaylor_13 2y agoHeck if your user system is just a Users table, you don’t even really need to consider build vs buy for them either. If you start doing it for sub-groups, hard agree but this is a space where it almost always pays dividends to roll your own first. The size of a company that needs to consider adding feature flags (versus one that already has them) is typically that in which building your own is quicker, cheaper, and most importantly: simpler.
- j45 2y agoGreat summary. Just starting with them and learning to improve your application of them is the best way to learn, too. There is one book on feature flags that had been written earlier, some of the independently published books by experienced tech folks out there are a goldmine. Feature Flags by Ben Nadel is one such book for me. There is an online version that is free as well. Happy to learn about others. https://featureflagsbook.com/ https://featureflagsbook.com/
- the_mitsuhiko 2y ago> I think there's a reasonable middle ground-point between having feature flags in a JSON file that you have to redeploy to change and using an (often expensive) feature flags as a service platform: roll your own simple system. The middle ground is a JSON file that is copied up and periodically refreshed. We (Sentry) moved from a managed software to just a YAML file with feature flags that is pushed to all containers. The benefit of just changing a file is that you have a lot of freedom of how you deal with it (eg: leave comments) and you have the history of who flipped it and for which reason.
- maccard 2y agoHow do you push the files to all of your containers? I’ve done this in the past with app specific endpoints but never found a solution I liked with containers.
- superb_dev 2y agoWe keep a JSON blob in Google Secret Manager for our flags. The service running in the container will reload the secret anytime it changes
- maccard 2y agoAh that’s a super nice feature. I’m mostly familiar with AWS who don’t have a neat way of doing this, you end up with a bespoke solution either with lambdas pushing to shared volumes or just polling s3 for updates.
- withinboredom 2y agoIf you are using kubernetes, you can mount the secret/ConfigMap as a volume and it will be updated automatically when changes occur. Then your application merely watches the file for updates.
- maccard 2y ago
- secondcoming 2y ago> feature flags in a JSON file that you have to redeploy to change Our config files are stored in their own repo. Pushes to the master branch trigger a Jenkins job that copies the config files to a GCP bucket. On startup, each machine pulls this config from GCS and everything just works. It's not a 'redeployment' in the sense that we don't push new images on each config change.
- selfmodruntime 2y agoWe do the same thing but slightly differently. If a new docker image is built, we deploy that image. If the config changes, an ansible job moves that config to the target host and the service is restartet with that new config file. Configs are mounted inside containers. It all runs on GitLab CI/CD.
- ljm 2y agoI think openfeature.dev is an attractive proposition these days - start off with an env-based provider or roll your own and if you get to a point where you need to buy, you only need to swap over a provider (or use a multi-provider).
- selfmodruntime 2y agoI like openfeature. Initially I thought that it was overengineered and quote honestly, I never grew to a size where I had to use anything but an env-provider paired with out CI/CD pipeline. But it gave me security that A/B testing would be possible if needed, and, more importantly, we had a unified API for feature flags and they were all defined in one place.