3 ms·

Something I love about the experiment tooling I've used at a few places now (Thread, Google) is the fact that state has typically been stored in source control.

Something I love about the experiment tooling I've used at a few places now (Thread, Google) is the fact that state has typically been stored in source control. i.e. not just the usage of flags in code, but the rollout/experiment definitions, the state of how much traffic is allocated to each branch, eligibility requirements, etc. This makes it easy to see what the current state is (without going to a UI), and also makes it much easier to build tooling around that state, as the API data sync problem disappears. Looking at DevCycle, it seems you've not taken this approach, is that right? Scanning your docs it seems there's an API to update state, but that fundamentally it's kept in your database, not in code. In my experience this isn't the best dev experience, so as dev experience is your USP, I'm interested in why you think this is a better experience, what benefits it can bring over a "git-ops" style, or what I've missed in your docs.

2023throwawayy 3y ago
Feature flags in source control seems to be missing the point. Separating feature flags and source let’s you decouple your code releases from your feature releases. Requiring a deploy of new to code to enable or disable a feature seems to negate almost all of the benefit of using feature flags to begin with.
- aleksiy123 3y ago
  They don't necessarily have to rolled out as part of the same release process. You can still decouple them without giving up them being version controlled which to me is also preferable over just toggling things through a gui. Having a release process for flags also allows you to run integration tests with those flags, canary alerting and automated rollbacks.
  - danpalmer 3y ago
    Yes this is how I view it. Just because it's in code doesn't mean it's the same codebase, same deployment process, same servers, or anything else. Code is the source of truth and log of how that truth changed over time.
jonathannorris 3y ago
Really interesting take on your experience with experimentation / feature flags and wanting it to be more stateful in code. I assume that follows the more Terraform-like infrastructure as code approach, which certainly makes sense to rollout infrastructure changes. We have a V1 of a Terraform provider where we hope to enable more control through terraform to manage infrastructure changes directly, but you are correct that has yet to be our core focus. Another way we hope to expose that state outside of our UI is building out a great CLI, which we are actively working on. Our primary usecase has been working with product development teams who have adopted Feature Flags as part of their workflows, using DevCycle to help them deploy features faster / safer to their end users, even deploying continuously as we do. Generally, customers integrate DevCycle into their Websites / Mobile Apps + API servers to control those feature deployments. We have had a couple of customers use us at the infrastructure layer. One of the best use cases I've seen is in a proxy service controlling the rollout of a new infrastructure stack. I'd love to dig into this use-case deeper and see how we can better support it in the future, "Ops" flags are one of the flag types we are looking to support. I would say that for customers looking to disconnect the deployment of code from the release of features, our approach where you can "release" features at any time has many advantages over "git-ops" style configurations. But certainly, for Ops use-cases where you are controlling infrastructure changes, we are believers in "git-ops" and use it ourselves. The challenge comes with how to connect those two deployment styles effectively.
- danpalmer 3y ago
  Thanks for the detailed reply, although I feel like I may have miscommunicated the aim here. I'm not really thinking about what is being launched, but how, and where the truth lives. A CLI is certainly a nice feature, but brings decisions that need to be made: who runs it, where do they run it, when, how do you know what was run, how do you deploy the CLI, and so on. The same can be said for, say, running your test suite – and the solution there is to have CI do it for you. Sure you can run, but the run that matters is the one when you merge your branch and that's done in a controlled environment typically defined in code. Rolling out features/flags is the same, and I think if the state of all the flags, features, traffic allocations, targeting, and so on, is all defined in code, then all those questions you get with a CLI go away. Who runs it? An automated process. Where do they run it? In a controlled environment, not on a dev machine. How do you know what was run? It's all there in code. How do you deploy the CLI? You don't need to. > for customers looking to disconnect the deployment of code from the release of features, our approach where you can "release" features at any time has many advantages over "git-ops" style configurations I'm interested as to why you say this has many advantages, because I don't see why a git based workflow couldn't run this. You could for example have DevCycle subscribe to the notifications for the git repo and update its internal state any time new changes are made to the git repo. That would be preferable over a UI or CLI because the whole state is there in a machine readable format ready for tooling to use.
esafak 3y ago
How did you represent the features flags in source control?
- danpalmer 3y ago
  Just a config file defining the flag, how it was targeted at users or other entities, how much it was rolled out, etc.

2023throwawayy 3y ago

Feature flags in source control seems to be missing the point. Separating feature flags and source let’s you decouple your code releases from your feature releases. Requiring a deploy of new to code to enable or disable a feature seems to negate almost all of the benefit of using feature flags to begin with.

aleksiy123 3y ago
They don't necessarily have to rolled out as part of the same release process. You can still decouple them without giving up them being version controlled which to me is also preferable over just toggling things through a gui. Having a release process for flags also allows you to run integration tests with those flags, canary alerting and automated rollbacks.
- danpalmer 3y ago
  Yes this is how I view it. Just because it's in code doesn't mean it's the same codebase, same deployment process, same servers, or anything else. Code is the source of truth and log of how that truth changed over time.

jonathannorris 3y ago

Really interesting take on your experience with experimentation / feature flags and wanting it to be more stateful in code. I assume that follows the more Terraform-like infrastructure as code approach, which certainly makes sense to rollout infrastructure changes. We have a V1 of a Terraform provider where we hope to enable more control through terraform to manage infrastructure changes directly, but you are correct that has yet to be our core focus. Another way we hope to expose that state outside of our UI is building out a great CLI, which we are actively working on. Our primary usecase has been working with product development teams who have adopted Feature Flags as part of their workflows, using DevCycle to help them deploy features faster / safer to their end users, even deploying continuously as we do. Generally, customers integrate DevCycle into their Websites / Mobile Apps + API servers to control those feature deployments. We have had a couple of customers use us at the infrastructure layer. One of the best use cases I've seen is in a proxy service controlling the rollout of a new infrastructure stack. I'd love to dig into this use-case deeper and see how we can better support it in the future, "Ops" flags are one of the flag types we are looking to support. I would say that for customers looking to disconnect the deployment of code from the release of features, our approach where you can "release" features at any time has many advantages over "git-ops" style configurations. But certainly, for Ops use-cases where you are controlling infrastructure changes, we are believers in "git-ops" and use it ourselves. The challenge comes with how to connect those two deployment styles effectively.

danpalmer 3y ago
Thanks for the detailed reply, although I feel like I may have miscommunicated the aim here. I'm not really thinking about what is being launched, but how, and where the truth lives. A CLI is certainly a nice feature, but brings decisions that need to be made: who runs it, where do they run it, when, how do you know what was run, how do you deploy the CLI, and so on. The same can be said for, say, running your test suite – and the solution there is to have CI do it for you. Sure you can run, but the run that matters is the one when you merge your branch and that's done in a controlled environment typically defined in code. Rolling out features/flags is the same, and I think if the state of all the flags, features, traffic allocations, targeting, and so on, is all defined in code, then all those questions you get with a CLI go away. Who runs it? An automated process. Where do they run it? In a controlled environment, not on a dev machine. How do you know what was run? It's all there in code. How do you deploy the CLI? You don't need to. > for customers looking to disconnect the deployment of code from the release of features, our approach where you can "release" features at any time has many advantages over "git-ops" style configurations I'm interested as to why you say this has many advantages, because I don't see why a git based workflow couldn't run this. You could for example have DevCycle subscribe to the notifications for the git repo and update its internal state any time new changes are made to the git repo. That would be preferable over a UI or CLI because the whole state is there in a machine readable format ready for tooling to use.

esafak 3y ago

How did you represent the features flags in source control?

danpalmer 3y ago
Just a config file defining the flag, how it was targeted at users or other entities, how much it was rolled out, etc.