4 ms·

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't thi

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allows simple rollback. Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback.

SpicyLemonZest 7y ago
Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the regional network, but only when deployed at scale” is a potential failure mode? Even if you could, how do you write a test for that?
- grey-area 7y ago
  Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for most of these outages now. The parent said: It’s not that the configuration change is “to blame”. Which I (and the article) disagree with.
  - SpicyLemonZest 7y ago
    Blaming the configuration system for outages is like blaming JIRA for bugs. Google and friends specifically try to ensure, as a matter of policy, that any change which might cause an outage is implemented as a config.
    - grey-area 7y ago
      From the article, the root cause was a configuration change: "In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions...Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage." I do think we should blame the configuration system, it is clearly not robust enough, not tested enough, and not resilient in case of failure - a bug in the config can bring down the system which manages the config and stop them fixing the original bug.
  - buckminster 7y ago
    So what happens when you need to update the control plane for configs? Do you add another layer?
    - grey-area 7y ago
      Who will configure the configurators? Hopefully this layer would be far more stable and very infrequently touched.
      - q3k 7y ago
        And something that's infrequently touched will likely be poorly understood and engineers will not have much knowledge on how to fix things fast when they break.
    - dboreham 7y ago
      The ultimate layer will be built from tobacco tins connected with tight wet string.
robszumski 7y ago
I am positive that Google has these systems in place. Or at least in many many places where config changes can go wrong. Hopefully they will share what caused it to fail in this circumstance.
klodolph 7y ago
> I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. That’s why I said the config change is “just the trigger”. Root cause analysis will generally result in multiple causes for any problem. > Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. Google already has that, you can see it in the postmortems for other outages. It’s called canary. > Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback. Unfortunately, in the real world config changes are hard to test. Not always, but often. Working on large deployments has taught me that even with config changes checked in to source control, with automatic canary and gradual rollouts, you will still have outages. Code doesn’t have 100% test coverage either. Chasing after 100% coverage is a pipe dream.
- grey-area 7y ago
  Unfortunately, in the real world config changes are hard to test. I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have been exposed as the new point of failure. That will require rethinking how configs are managed and how they are applied. I'm not talking about 100% test coverage, but failure recovery. The article does suggest that config was the root cause: In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region What I'm suggesting is that what google (and Amazon) has for configs is not good enough, that the root cause of this outage was in fact a config change (like all the others), and that what is required is a rethink of configs which recognises that they need an entirely separate control plane, should never be global, should not be hard to test etc. Clearly here, since the bad config was able to stop them actually fixing the problem, they need to rethink how their configs are applied somehow. As this keeps happening with different config changes I'd suggest this is not a one-off isolated problem but a symptom of a broader failure to tackle the fragile nature of current config systems.
remus 7y ago
> Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. > Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback. Google already do this. The SRE book goes in to details https://landing.google.com/sre/books/ https://landing.google.com/sre/books/

SpicyLemonZest 7y ago

Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the regional network, but only when deployed at scale” is a potential failure mode? Even if you could, how do you write a test for that?

grey-area 7y ago
Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for most of these outages now. The parent said: It’s not that the configuration change is “to blame”. Which I (and the article) disagree with.
- SpicyLemonZest 7y ago
  Blaming the configuration system for outages is like blaming JIRA for bugs. Google and friends specifically try to ensure, as a matter of policy, that any change which might cause an outage is implemented as a config.
  - grey-area 7y ago
    From the article, the root cause was a configuration change: "In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions...Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage." I do think we should blame the configuration system, it is clearly not robust enough, not tested enough, and not resilient in case of failure - a bug in the config can bring down the system which manages the config and stop them fixing the original bug.
- buckminster 7y ago
  So what happens when you need to update the control plane for configs? Do you add another layer?
  - grey-area 7y ago
    Who will configure the configurators? Hopefully this layer would be far more stable and very infrequently touched.
    - q3k 7y ago
      And something that's infrequently touched will likely be poorly understood and engineers will not have much knowledge on how to fix things fast when they break.
  - dboreham 7y ago
    The ultimate layer will be built from tobacco tins connected with tight wet string.

robszumski 7y ago

I am positive that Google has these systems in place. Or at least in many many places where config changes can go wrong. Hopefully they will share what caused it to fail in this circumstance.

klodolph 7y ago

> I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. That’s why I said the config change is “just the trigger”. Root cause analysis will generally result in multiple causes for any problem. > Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. Google already has that, you can see it in the postmortems for other outages. It’s called canary. > Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback. Unfortunately, in the real world config changes are hard to test. Not always, but often. Working on large deployments has taught me that even with config changes checked in to source control, with automatic canary and gradual rollouts, you will still have outages. Code doesn’t have 100% test coverage either. Chasing after 100% coverage is a pipe dream.

grey-area 7y ago
Unfortunately, in the real world config changes are hard to test. I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have been exposed as the new point of failure. That will require rethinking how configs are managed and how they are applied. I'm not talking about 100% test coverage, but failure recovery. The article does suggest that config was the root cause: In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region What I'm suggesting is that what google (and Amazon) has for configs is not good enough, that the root cause of this outage was in fact a config change (like all the others), and that what is required is a rethink of configs which recognises that they need an entirely separate control plane, should never be global, should not be hard to test etc. Clearly here, since the bad config was able to stop them actually fixing the problem, they need to rethink how their configs are applied somehow. As this keeps happening with different config changes I'd suggest this is not a one-off isolated problem but a symptom of a broader failure to tackle the fragile nature of current config systems.

remus 7y ago

> Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. > Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback. Google already do this. The SRE book goes in to details https://landing.google.com/sre/books/ https://landing.google.com/sre/books/