5 ms·

So... you're 50ms away from a total global outage and/or major feature fault? Why would I want that? How do I control it for green-blue? For rational isolation

So... you're 50ms away from a total global outage and/or major feature fault? Why would I want that? How do I control it for green-blue? For rational isolation? Is there native support for canary releases?

coldtea 7y ago
Yeah, because speed of deployment prevents CI/tests/checks /s (Seriously, how do people come with such facile counter-arguments?)
- KirinDave 7y ago
  How does CI actually prevent these kinds of outages? Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Do you think those organizations don't spend enough energy for the average SV commute on test runs per release? Are you seriously going to pretend that those efforts "solve" reliability and stability in any non-trivial system? It's actually not hard at all to write fast software deployment. Many people do it accidentally. But to tie it directly to editors and stovepipe the infrastructure around testing, that seems to me like a great business model that any rational engineer interested in reliability should immediately reject unless it's coupled with open source, statistically valid canary tooling at the bare minimum. What you actually want at that degree of universal knife-switching is rollbacks.
  - penagwin 7y ago
    > Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?" Just because your tests/ci/etc. can't catch every single issue that could go wrong doesn't make them useless. Just because people die every single day in car accidents doesn't mean car's safety features are useless. You're correct in that a CI should not in any way be considered a "perfect solution for outages", but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful.
    - KirinDave 7y ago
      > This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?" Actually, your refutation is precisely what I'm trying to say! The GP implied that CI and UTs fix these problems. They don't. In the same sense that seatbelts and air bags may reduce the chance of a fatal accident, they do not replace safe driving, well-trained drivers and intelligent infrastructure design. > but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful. I didn't say otherwise. In fact I agreed those measures should be in place. They're just not sufficient. You will still face bad releases even if you do this, and so you shouldn't do fleetwide rollouts unless you just don't care about your system falling apart. Please don't misrepresent my comment.
  - coldtea 7y ago
    >Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Yes, multiple times. It just didn't help with each and every outage, which is a different thing. For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot. Besides the concern about outages is totally unrelated to "speed of deployment". Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server).
    - KirinDave 7y ago
      > Yes, multiple times. It just didn't help with each and every outage, which is a different thing. So then maybe the idea that a prompt global rollout is a bad one even if you have good tests and coverage. > For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot. I'm here to tell you, as someone who has been at this for a long time and someone who is actually doing SRE at Google at the outer layers: No. It matters. We do canaries because they let you find problems no reasonable person cold catch with tests. Tests themselves are only cone component of a chain of reliability best practices, starting at coding practice and expressive type systems with static checks and moving outwards to reploying the code carefully and with intentionality. > Besides the concern about outages is totally unrelated to "speed of deployment". This is obviously false. Slower rollouts of new software and canaries are a proven way to reduce the impact of a bad push. Once your system gets modestly complicated, it can be very difficult to spot a bad push with naive unit testing which (let's be real here) often just mocks out dependencies anyways. > Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server). Right, but very fast deployment to client machines is not hard. The CD model is just inherently dangerous unless it has automated safeguards against the complexities of even a modestly large production app. Personally, if your staging environment is CD I think that's great (up until your org gets big enough that a broken build pisses off other devs). But to prod? I will resign before I let you do it. Everyone who does software at scale agrees it's not a good plan to do CD on prod. There are whole books about why you shouldn't do it.
toast0 7y ago
> So... you're 50ms away from a total global outage and/or major feature fault? And 50ms away from restoring service. The degree of care that needs to be put into a release is related to the potential damage a bad release can cause and the time to repair. Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was). If you're building NES cartridges, you should really get it right the first time, since there's no way to update. If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly.
- KirinDave 7y ago
  > And 50ms away from restoring service. Sure, but fast rollbacks are much easier than fast rollouts. > Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was). A uniform fast rollout is almost always worse, and a separate problem from data recovery or fast rollbacks. But I agree with this paragraph in principle. > If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly. When you're doing more than basic presentation of semi-dynamic content, most CSPs make multi-region deployment very affordable and easy. It's generally a failure of early engineering departments or an accumulation of technical debt that makes it otherwise in 2019. Too many folks assume it is something they should do later rather than get right to start. Canaries and regional redundancy should be part of every product's basic launch checklist, imo. People love to say any kind of software engineering best practice is bad, because they love to pretend software engineering has no value. But for every flash UX trick or tracking feature someone ships providing value, that's completely overshadowed by being down when a user needs you. Reliability not only saves you time at the developer level by making features more predictable, but it means you need to worry slightly less about funnel optimization because your user attrition is lower.
kreetx 7y ago
Isn't this better than deploying for 15 minutes, having an outage then, and then rolling back?
- KirinDave 7y ago
  Not if that 15 minute deploy rolled out over a canary that gives me much less of a chance to actually have outages, yeah. Knife-switch rollbacks are good though, tbf. I just don't think that's particularly challenging.
pbiggar 7y ago
Author here. Was this in response to the article or the title? Because the article goes into great detail about this. To answer, the whole system is based on feature flags (which are equivalent in this case to green-blue/canary releases). So yes, there is a massive amount of control for this; in fact, that's the whole point of the article.
- KirinDave 7y ago
  Feature flags are not canaries. I checked the article twice, even did a find on the text of the view source because sometimes I miss stuff. You just didn't talk about the actual complexity of deployment. And that's your call, but as you can see from this thread there are people who think CI/CD to prod with no controls is a good call.
  - telotortium 7y ago
    (Also an SRE at Google) From my reading of the article, the way that Dark is intended to work is that the language and environment enforces that changes to code accessible in production are forbidden to be made without putting the change behind a feature flag. Feature flags can be written so that flag flips are rolled out gradually - to whitelisted users, percentage of users, etc. (I'd be interested in how sophisticated the set of requests enabled for the feature flags can be specified.) Writing a tool to instantly revert the feature flags to their state at a previous timestamp is straightforward in principle. And presumably there'd be no need to write separate configuration files in a pure Dark environment, so changes to configuration would also be feature flagged. In this case, there would be in theory no need for a separate canary process, since the feature flags themselves can be canaried. I just hope that Dark will encourage teams to canary changes rather than have "easy" defaults of "cowboy deployment to prod". Dark would probably make it very easy to roll back the simple changes that result in exceptions being thrown on every request. All the most interesting outages that I've seen have involved emergent interactions that put the system into a state unrecoverable without manual intervention. It's unclear how Dark would help with that from my reading.
    - 7y ago
cbetti 7y ago
From what I gather, Dark's deployment "unit" is the entire AST covered by a function, and these deployment units are versioned and promotable via an independent criteria. Perhaps one of the promotion strategies could involve a human controlled criteria. I would like to understand their plans on this point a bit better as well. I see immense power in switching a promotion strategy away from large units, like entire containers or versioned artifacts, to fine grained components, like versioned functions. Do you see any advantages here for improving your platform's availability?

coldtea 7y ago

Yeah, because speed of deployment prevents CI/tests/checks /s (Seriously, how do people come with such facile counter-arguments?)

KirinDave 7y ago
How does CI actually prevent these kinds of outages? Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Do you think those organizations don't spend enough energy for the average SV commute on test runs per release? Are you seriously going to pretend that those efforts "solve" reliability and stability in any non-trivial system? It's actually not hard at all to write fast software deployment. Many people do it accidentally. But to tie it directly to editors and stovepipe the infrastructure around testing, that seems to me like a great business model that any rational engineer interested in reliability should immediately reject unless it's coupled with open source, statistically valid canary tooling at the bare minimum. What you actually want at that degree of universal knife-switching is rollbacks.
- penagwin 7y ago
  > Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?" Just because your tests/ci/etc. can't catch every single issue that could go wrong doesn't make them useless. Just because people die every single day in car accidents doesn't mean car's safety features are useless. You're correct in that a CI should not in any way be considered a "perfect solution for outages", but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful.
  - KirinDave 7y ago
    > This is not a fair statement. This is like saying "Did seatbelts save the victims who died in car accidents X/Y/Z?" Actually, your refutation is precisely what I'm trying to say! The GP implied that CI and UTs fix these problems. They don't. In the same sense that seatbelts and air bags may reduce the chance of a fatal accident, they do not replace safe driving, well-trained drivers and intelligent infrastructure design. > but an automated system that tests, deploys, etc. in a predictable and (again) testable manor is at least helpful. I didn't say otherwise. In fact I agreed those measures should be in place. They're just not sufficient. You will still face bad releases even if you do this, and so you shouldn't do fleetwide rollouts unless you just don't care about your system falling apart. Please don't misrepresent my comment.
- coldtea 7y ago
  >Did CI save Google from the networking outage or tests prevent a global Cloudflare outage? Yes, multiple times. It just didn't help with each and every outage, which is a different thing. For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot. Besides the concern about outages is totally unrelated to "speed of deployment". Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server).
  - KirinDave 7y ago
    > Yes, multiple times. It just didn't help with each and every outage, which is a different thing. So then maybe the idea that a prompt global rollout is a bad one even if you have good tests and coverage. > For the ones not preventable by CI (e.g. a disk was borked, power was down, but no integration / programming error etc a CI process could possibly spot) the question is moot. I'm here to tell you, as someone who has been at this for a long time and someone who is actually doing SRE at Google at the outer layers: No. It matters. We do canaries because they let you find problems no reasonable person cold catch with tests. Tests themselves are only cone component of a chain of reliability best practices, starting at coding practice and expressive type systems with static checks and moving outwards to reploying the code carefully and with intentionality. > Besides the concern about outages is totally unrelated to "speed of deployment". This is obviously false. Slower rollouts of new software and canaries are a proven way to reduce the impact of a bad push. Once your system gets modestly complicated, it can be very difficult to spot a bad push with naive unit testing which (let's be real here) often just mocks out dependencies anyways. > Increased speed of deployment helps testing and iteration (obviously faster deployment speeds also means faster deployment to testing and staging environments, not just to the final server). Right, but very fast deployment to client machines is not hard. The CD model is just inherently dangerous unless it has automated safeguards against the complexities of even a modestly large production app. Personally, if your staging environment is CD I think that's great (up until your org gets big enough that a broken build pisses off other devs). But to prod? I will resign before I let you do it. Everyone who does software at scale agrees it's not a good plan to do CD on prod. There are whole books about why you shouldn't do it.

toast0 7y ago

> So... you're 50ms away from a total global outage and/or major feature fault? And 50ms away from restoring service. The degree of care that needs to be put into a release is related to the potential damage a bad release can cause and the time to repair. Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was). If you're building NES cartridges, you should really get it right the first time, since there's no way to update. If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly.

KirinDave 7y ago
> And 50ms away from restoring service. Sure, but fast rollbacks are much easier than fast rollouts. > Reducing time to deploy reduces time to repair for most errors (certainly not for errors than write bad states, if you write bad data to your database, that's most likely a long repair regardless of how fast to deploy it was). A uniform fast rollout is almost always worse, and a separate problem from data recovery or fast rollbacks. But I agree with this paragraph in principle. > If you're making a low cost internet service, for small changes, it's not unreasonable to try things without a lot of testing before hand as long as you commit to detecting and rolling back quickly. When you're doing more than basic presentation of semi-dynamic content, most CSPs make multi-region deployment very affordable and easy. It's generally a failure of early engineering departments or an accumulation of technical debt that makes it otherwise in 2019. Too many folks assume it is something they should do later rather than get right to start. Canaries and regional redundancy should be part of every product's basic launch checklist, imo. People love to say any kind of software engineering best practice is bad, because they love to pretend software engineering has no value. But for every flash UX trick or tracking feature someone ships providing value, that's completely overshadowed by being down when a user needs you. Reliability not only saves you time at the developer level by making features more predictable, but it means you need to worry slightly less about funnel optimization because your user attrition is lower.

kreetx 7y ago

Isn't this better than deploying for 15 minutes, having an outage then, and then rolling back?

KirinDave 7y ago
Not if that 15 minute deploy rolled out over a canary that gives me much less of a chance to actually have outages, yeah. Knife-switch rollbacks are good though, tbf. I just don't think that's particularly challenging.

pbiggar 7y ago

Author here. Was this in response to the article or the title? Because the article goes into great detail about this. To answer, the whole system is based on feature flags (which are equivalent in this case to green-blue/canary releases). So yes, there is a massive amount of control for this; in fact, that's the whole point of the article.

KirinDave 7y ago
Feature flags are not canaries. I checked the article twice, even did a find on the text of the view source because sometimes I miss stuff. You just didn't talk about the actual complexity of deployment. And that's your call, but as you can see from this thread there are people who think CI/CD to prod with no controls is a good call.
- telotortium 7y ago
  (Also an SRE at Google) From my reading of the article, the way that Dark is intended to work is that the language and environment enforces that changes to code accessible in production are forbidden to be made without putting the change behind a feature flag. Feature flags can be written so that flag flips are rolled out gradually - to whitelisted users, percentage of users, etc. (I'd be interested in how sophisticated the set of requests enabled for the feature flags can be specified.) Writing a tool to instantly revert the feature flags to their state at a previous timestamp is straightforward in principle. And presumably there'd be no need to write separate configuration files in a pure Dark environment, so changes to configuration would also be feature flagged. In this case, there would be in theory no need for a separate canary process, since the feature flags themselves can be canaried. I just hope that Dark will encourage teams to canary changes rather than have "easy" defaults of "cowboy deployment to prod". Dark would probably make it very easy to roll back the simple changes that result in exceptions being thrown on every request. All the most interesting outages that I've seen have involved emergent interactions that put the system into a state unrecoverable without manual intervention. It's unclear how Dark would help with that from my reading.
  - 7y ago

cbetti 7y ago

From what I gather, Dark's deployment "unit" is the entire AST covered by a function, and these deployment units are versioned and promotable via an independent criteria. Perhaps one of the promotion strategies could involve a human controlled criteria. I would like to understand their plans on this point a bit better as well. I see immense power in switching a promotion strategy away from large units, like entire containers or versioned artifacts, to fine grained components, like versioned functions. Do you see any advantages here for improving your platform's availability?