3 ms·

Author here. Was this in response to the article or the title? Because the article goes into great detail about this. To answer, the whole system is based on f

Author here. Was this in response to the article or the title? Because the article goes into great detail about this. To answer, the whole system is based on feature flags (which are equivalent in this case to green-blue/canary releases). So yes, there is a massive amount of control for this; in fact, that's the whole point of the article.

KirinDave 7y ago
Feature flags are not canaries. I checked the article twice, even did a find on the text of the view source because sometimes I miss stuff. You just didn't talk about the actual complexity of deployment. And that's your call, but as you can see from this thread there are people who think CI/CD to prod with no controls is a good call.
- telotortium 7y ago
  (Also an SRE at Google) From my reading of the article, the way that Dark is intended to work is that the language and environment enforces that changes to code accessible in production are forbidden to be made without putting the change behind a feature flag. Feature flags can be written so that flag flips are rolled out gradually - to whitelisted users, percentage of users, etc. (I'd be interested in how sophisticated the set of requests enabled for the feature flags can be specified.) Writing a tool to instantly revert the feature flags to their state at a previous timestamp is straightforward in principle. And presumably there'd be no need to write separate configuration files in a pure Dark environment, so changes to configuration would also be feature flagged. In this case, there would be in theory no need for a separate canary process, since the feature flags themselves can be canaried. I just hope that Dark will encourage teams to canary changes rather than have "easy" defaults of "cowboy deployment to prod". Dark would probably make it very easy to roll back the simple changes that result in exceptions being thrown on every request. All the most interesting outages that I've seen have involved emergent interactions that put the system into a state unrecoverable without manual intervention. It's unclear how Dark would help with that from my reading.
  - pbiggar 7y ago
    Thank you! Yes, your understanding is correct and your explanation is lovely. Much appreciated. A thing to clarify is that you're not really deploying code in 50ms. It's more that all the code you write is in production, behind a feature flag. But, you cant change unflagged code and you also cant change the old code, you can only set a flag that some users see the new code.

KirinDave 7y ago

Feature flags are not canaries. I checked the article twice, even did a find on the text of the view source because sometimes I miss stuff. You just didn't talk about the actual complexity of deployment. And that's your call, but as you can see from this thread there are people who think CI/CD to prod with no controls is a good call.

telotortium 7y ago
(Also an SRE at Google) From my reading of the article, the way that Dark is intended to work is that the language and environment enforces that changes to code accessible in production are forbidden to be made without putting the change behind a feature flag. Feature flags can be written so that flag flips are rolled out gradually - to whitelisted users, percentage of users, etc. (I'd be interested in how sophisticated the set of requests enabled for the feature flags can be specified.) Writing a tool to instantly revert the feature flags to their state at a previous timestamp is straightforward in principle. And presumably there'd be no need to write separate configuration files in a pure Dark environment, so changes to configuration would also be feature flagged. In this case, there would be in theory no need for a separate canary process, since the feature flags themselves can be canaried. I just hope that Dark will encourage teams to canary changes rather than have "easy" defaults of "cowboy deployment to prod". Dark would probably make it very easy to roll back the simple changes that result in exceptions being thrown on every request. All the most interesting outages that I've seen have involved emergent interactions that put the system into a state unrecoverable without manual intervention. It's unclear how Dark would help with that from my reading.
- pbiggar 7y ago
  Thank you! Yes, your understanding is correct and your explanation is lovely. Much appreciated. A thing to clarify is that you're not really deploying code in 50ms. It's more that all the code you write is in production, behind a feature flag. But, you cant change unflagged code and you also cant change the old code, you can only set a flag that some users see the new code.