3 ms·

> [A] configuration change [...] was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of

> [A] configuration change [...] was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... > Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage. Someone forgot to classify management traffic as high-priority? Oops. The description is vague about what devices ("servers") were misconfigured. Did someone tell all google service pods in the affected regions to restrict bandwidth by over 50%? Mentioning "server" and then talking about network congestion is confusing. How would restricted bandwidth utilization on servers cause network congestion, unless load balancers saturated the network by re-sending requests to servers because none of them were responding?

singron 7y ago
> The description is vague about what devices ("servers") were misconfigured. "servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details. > How would restricted bandwidth utilization on servers cause network congestion... This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100 qps of traffic between equally sized pods A and B. If each pod has an actual capacity of 60 qps and received 50 qps, then everything is fine. However, if you configure your load balancer not to send more than 10 qps to A, then it has to send the remaining 90 qps to B. Now B is actually overloaded by 50%. Using automatic utilization based load balancing can prevent this in some cases, but it can also cause it if utilization isn't reported accurately. > Someone forgot to classify management traffic as high-priority? Oops. I have some sympathy. During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic. If you have extreme foresight, maybe you ignored that temptation or built in an escape hatch for emergencies. However, with a complicated layered infrastructure, it's very difficult to be sure that all network communication has the appropriate network priority, and you usually don't find out until a situation like this.
- laurentl 7y ago
  > During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic Honest question: is it not best practice to have an isolated, dedicated management network? I can’t for the life of me understand why a misconfig on the production network should hamper access through the admin network. Unless on Google’s scale it’s not the proper way to design and operate a network ?
  - remus 7y ago
    Presumably it's a trade off of complexity against redundancy, and at the scale that google's datacenters run the complexity is too high to make it worthwhile.
  - deleted 7y ago
    [deleted]
  - SpicyLemonZest 7y ago
    On Google's scale, the networks are themselves production systems. So the question they face isn't whether to keep a single isolated network, but how long it's worth keeping the recursion going.
xtracto 7y ago
A colleague mine at some point used some Ansible command within AWS which instructed AWS to terminate instances that did not have a specific tag... we were so scared as we saw production instances losing connection one by one, until we realized what was happening. Fortunately, we had set the important instances to have termination protection. But man, the kind of damage you can do with a single command is huge.

singron 7y ago

> The description is vague about what devices ("servers") were misconfigured. "servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details. > How would restricted bandwidth utilization on servers cause network congestion... This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100 qps of traffic between equally sized pods A and B. If each pod has an actual capacity of 60 qps and received 50 qps, then everything is fine. However, if you configure your load balancer not to send more than 10 qps to A, then it has to send the remaining 90 qps to B. Now B is actually overloaded by 50%. Using automatic utilization based load balancing can prevent this in some cases, but it can also cause it if utilization isn't reported accurately. > Someone forgot to classify management traffic as high-priority? Oops. I have some sympathy. During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic. If you have extreme foresight, maybe you ignored that temptation or built in an escape hatch for emergencies. However, with a complicated layered infrastructure, it's very difficult to be sure that all network communication has the appropriate network priority, and you usually don't find out until a situation like this.

laurentl 7y ago
> During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic Honest question: is it not best practice to have an isolated, dedicated management network? I can’t for the life of me understand why a misconfig on the production network should hamper access through the admin network. Unless on Google’s scale it’s not the proper way to design and operate a network ?
- remus 7y ago
  Presumably it's a trade off of complexity against redundancy, and at the scale that google's datacenters run the complexity is too high to make it worthwhile.
- deleted 7y ago
  [deleted]
- SpicyLemonZest 7y ago
  On Google's scale, the networks are themselves production systems. So the question they face isn't whether to keep a single isolated network, but how long it's worth keeping the recursion going.

xtracto 7y ago

A colleague mine at some point used some Ansible command within AWS which instructed AWS to terminate instances that did not have a specific tag... we were so scared as we saw production instances losing connection one by one, until we realized what was happening. Fortunately, we had set the important instances to have termination protection. But man, the kind of damage you can do with a single command is huge.