7 ms·

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it m

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

summerlight 7y ago
Typically configuration changes tend to have much more impacts compared to implementation codes at the same line counts hence its consequences are less predictable. Also, it usually coordinates multiple systems at once, so it is much harder to test especially when it comes to a capacity problem caused by complex interactions among multiple systems. Canary or gradual roll-out cannot reliably catch this class of problems, since it's not really visible in a small scale.
sidibe 7y ago
It is harder to make useful tests for configuration changes than other kinds of changes.
- shereadsthenews 7y ago
  That's not true. Configuration is just another input to a system and the outputs can be checked. There is nothing difficult about testing configs that is not already difficult about testing code.
  - joshuamorton 7y ago
    There is: many configuration changes are, by nature, global. Code changes can be isolated and unit tested. Config changes often can't be. You can still canary them, usually, but you lose some protection.
  - keypusher 7y ago
    No, configuration defines the system and is therefore significantly harder to test. Application code can be unit tested and run in staging against simulated real world traffic. Configuration, on the other hand, often differs so much between production and staging that the only real way to test is to roll it out gradually and monitor the results. Just make staging and production identical then right? Easier said than done when your production infrastructure runs most of the worlds internet traffic. There’s a good reason that the last major outages to both AWS and Google were caused by infrastructure configuration changes, and it isn’t that their engineering sucks.
klodolph 7y ago
A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an incident like this, you would typically review why the code allowed this unintended configuration change, and change the code to protect against this kind of misconfiguration in the future. The problem is that when you ask, “why?” you can end up with multiple different answers depending on how you answer the question. Configuration changes are also somewhat difficult to test.
- grey-area 7y ago
  The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allows simple rollback. Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback.
  - SpicyLemonZest 7y ago
    Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the regional network, but only when deployed at scale” is a potential failure mode? Even if you could, how do you write a test for that?
    - grey-area 7y ago
wilde 7y ago
Configuration often goes through less testing and scrutiny, while simultaneously having a significantly faster deployment process.
flukus 7y ago
Well you can't just change code, it has to get reviewed, it has to go to QA, it has to go to UAT, it has to get signed off in triplicate by all the major stake holders. Configuration changes are easy though, they don't have to go through all these error prevention steps, we can just have our less technical support staff make configuration changes live in production. In fact we'll build them a DSL so we never have to make urgent code changes again...
- HALtheWise 7y ago
  I guarantee that Google does not allow non-technical support staff to make configuration changes to the core routing infrastructure of their datacenters. Other places might, but they run a much tighter ship than that.
- keypusher 7y ago
  A modern ops team should be sending configuration changes through the same code review process application code goes through. Infrastructure changes in something like Terraform, configuration changes in Puppet/Chef/etc, these are deployed and tested in dev, staging, prod.
fulafel 7y ago
I don't know if there is a science in "root cause analysis" to pickin the one to put in the headline? In this case, like most similar ones, it seems obvious that there are many things that conspired together to mess things up. If I got to decide based on the description, I would point the finger at the system properties that caused the 5 hour delay to reconfigure the network capacity.
peterwwillis 7y ago
Almost all outages are due to changes. Bugs in code usually surface pretty quickly, and hardware issues don't pose a problem in such large-scale infrastructure. Changes causing outages is par for the course. That's why the industry standard is now Infrastructure-as-Code and Immutable Infrastructure: you try to prevent anything from changing in a way that will later break, and manage it in ways that will be easier to fix when it does break. Tools like Terraform are popular today because they allow the planning and staging of changes across complex services. They're still pretty limited, but mapping out dependencies and simulating changes can surface errors before you run into them, thus making it less necessary to perform a rollback. But unexpected problems still happen, which is why you need to test your rollbacks, and intentionally stress random parts of your system to discover unknown bottlenecks. Part of the purpose for stress testing is to have a realistic idea of what kind of capacity you will really have under different conditions. But it's also nearly impossible to accurately stress test production systems without consequences. There are ways for Google to look for change issues, and they probably have lots of safeguards in place, but we don't know what they actually do to test changes. Some of their postmortems have pointed at a lack of stringent change control procedures. Hopefully they will practice what they preach (open/blameless postmortems) and share more details soon.
fowl2 7y ago
In a well designed system I guess this is the only thing that can go wrong.
- deleted 7y ago
  [deleted]
singron 7y ago
It's common practice in many organizations to effectively deploy new code with configuration changes. E.g. you write new code that is initially disabled then enable it with a configuration change. Since the code deploy didn't fail, you get a false confidence in the code and the configuration change seems like a safe "flag flip".
- joshuamorton 7y ago
  I would phrase this a bit differently: Binary version changes are a special case of configuration change that we (swes?) are particularly adept at managing reliably and safely. But there are lots of other config changes that are potentially dangerous, and that we aren't as good at doing safely.
chippy 7y ago
All infrastructure these days are configurations rather than code. K8s is basically all a set of config files, even a dockerfile is a config file.
Eridrus 7y ago
As others have mentioned: testing configuration changes is very hard. Why is it very hard? Because when you are the size of Google, there is no second version of prod to test things in, so the usual software engineering solution of trying the new thing in isolation and checking if it worked is unrealistic. I think these constant failures from config changes should cause folks to re-evaluate how they do config changes though. If we can't just do a green/blue deploy of config changes like this, we probably need some other solution, whether it be the watchdog timers mentioned elsewhere in the thread, or some system that is able to show you the impact of a config change before it takes effect (probably more realistic for single services such as networking, rather than all config changes).

summerlight 7y ago

Typically configuration changes tend to have much more impacts compared to implementation codes at the same line counts hence its consequences are less predictable. Also, it usually coordinates multiple systems at once, so it is much harder to test especially when it comes to a capacity problem caused by complex interactions among multiple systems. Canary or gradual roll-out cannot reliably catch this class of problems, since it's not really visible in a small scale.

sidibe 7y ago

It is harder to make useful tests for configuration changes than other kinds of changes.

shereadsthenews 7y ago
That's not true. Configuration is just another input to a system and the outputs can be checked. There is nothing difficult about testing configs that is not already difficult about testing code.
- joshuamorton 7y ago
  There is: many configuration changes are, by nature, global. Code changes can be isolated and unit tested. Config changes often can't be. You can still canary them, usually, but you lose some protection.
- keypusher 7y ago
  No, configuration defines the system and is therefore significantly harder to test. Application code can be unit tested and run in staging against simulated real world traffic. Configuration, on the other hand, often differs so much between production and staging that the only real way to test is to roll it out gradually and monitor the results. Just make staging and production identical then right? Easier said than done when your production infrastructure runs most of the worlds internet traffic. There’s a good reason that the last major outages to both AWS and Google were caused by infrastructure configuration changes, and it isn’t that their engineering sucks.

klodolph 7y ago

A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an incident like this, you would typically review why the code allowed this unintended configuration change, and change the code to protect against this kind of misconfiguration in the future. The problem is that when you ask, “why?” you can end up with multiple different answers depending on how you answer the question. Configuration changes are also somewhat difficult to test.

grey-area 7y ago
The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allows simple rollback. Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback.
- SpicyLemonZest 7y ago
  Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the regional network, but only when deployed at scale” is a potential failure mode? Even if you could, how do you write a test for that?
  - grey-area 7y ago

wilde 7y ago

Configuration often goes through less testing and scrutiny, while simultaneously having a significantly faster deployment process.

flukus 7y ago

Well you can't just change code, it has to get reviewed, it has to go to QA, it has to go to UAT, it has to get signed off in triplicate by all the major stake holders. Configuration changes are easy though, they don't have to go through all these error prevention steps, we can just have our less technical support staff make configuration changes live in production. In fact we'll build them a DSL so we never have to make urgent code changes again...

HALtheWise 7y ago
I guarantee that Google does not allow non-technical support staff to make configuration changes to the core routing infrastructure of their datacenters. Other places might, but they run a much tighter ship than that.
keypusher 7y ago
A modern ops team should be sending configuration changes through the same code review process application code goes through. Infrastructure changes in something like Terraform, configuration changes in Puppet/Chef/etc, these are deployed and tested in dev, staging, prod.

fulafel 7y ago

I don't know if there is a science in "root cause analysis" to pickin the one to put in the headline? In this case, like most similar ones, it seems obvious that there are many things that conspired together to mess things up. If I got to decide based on the description, I would point the finger at the system properties that caused the 5 hour delay to reconfigure the network capacity.

peterwwillis 7y ago

Almost all outages are due to changes. Bugs in code usually surface pretty quickly, and hardware issues don't pose a problem in such large-scale infrastructure. Changes causing outages is par for the course. That's why the industry standard is now Infrastructure-as-Code and Immutable Infrastructure: you try to prevent anything from changing in a way that will later break, and manage it in ways that will be easier to fix when it does break. Tools like Terraform are popular today because they allow the planning and staging of changes across complex services. They're still pretty limited, but mapping out dependencies and simulating changes can surface errors before you run into them, thus making it less necessary to perform a rollback. But unexpected problems still happen, which is why you need to test your rollbacks, and intentionally stress random parts of your system to discover unknown bottlenecks. Part of the purpose for stress testing is to have a realistic idea of what kind of capacity you will really have under different conditions. But it's also nearly impossible to accurately stress test production systems without consequences. There are ways for Google to look for change issues, and they probably have lots of safeguards in place, but we don't know what they actually do to test changes. Some of their postmortems have pointed at a lack of stringent change control procedures. Hopefully they will practice what they preach (open/blameless postmortems) and share more details soon.

fowl2 7y ago

In a well designed system I guess this is the only thing that can go wrong.

deleted 7y ago
[deleted]

singron 7y ago

It's common practice in many organizations to effectively deploy new code with configuration changes. E.g. you write new code that is initially disabled then enable it with a configuration change. Since the code deploy didn't fail, you get a false confidence in the code and the configuration change seems like a safe "flag flip".

joshuamorton 7y ago
I would phrase this a bit differently: Binary version changes are a special case of configuration change that we (swes?) are particularly adept at managing reliably and safely. But there are lots of other config changes that are potentially dangerous, and that we aren't as good at doing safely.

chippy 7y ago

All infrastructure these days are configurations rather than code. K8s is basically all a set of config files, even a dockerfile is a config file.

Eridrus 7y ago

As others have mentioned: testing configuration changes is very hard. Why is it very hard? Because when you are the size of Google, there is no second version of prod to test things in, so the usual software engineering solution of trying the new thing in isolation and checking if it worked is unrealistic. I think these constant failures from config changes should cause folks to re-evaluate how they do config changes though. If we can't just do a green/blue deploy of config changes like this, we probably need some other solution, whether it be the watchdog timers mentioned elsewhere in the thread, or some system that is able to show you the impact of a config change before it takes effect (probably more realistic for single services such as networking, rather than all config changes).