4 ms·

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration c

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?! I'm all for automation, but WTF? Insert even a semi-competent engineer in the loop to monitor the configuration change as it propagates around and the entire problem could have been addressed almost trivially, as the human engineers eventually decided to do.

jvolkman 11y ago
The human part of any process will also eventually fail, and it's much more difficult to fix human bugs. Better to shoot for full automation.
heisnotanalien 11y ago
But a computer could do the same thing. It would be possible to alert on all the routes being blown away?
DanielDent 11y ago
Any sufficiently large system quickly reaches a point where a human has difficulty tracking what the system should look like. Google has at least tens of data center locations, each of which will have multiple physical failure domains. There are also many discontiguous routes being announced at all of their network PoPs. They have substantially more PoPs than data centers. It very quickly gets too much to reasonably expect people to be able to keep track of what the system should look like, let alone grasping what it does look like.
dsl 11y ago
First of all, BGP is core to Google's load balancing architecture. So within a single datacenter you probably have at least a few dozen devices down stream from each edge router. Secondly, I'm seeing just shy of 500 individual prefixes, 282 directly connected peers (other networks), and a presence at over 100 physical internet exchanges, just for one of Google's four ASes. Would you be able to read over that configuration and tell me if it has errors?
- deleted 11y ago
  [deleted]

jvolkman 11y ago

The human part of any process will also eventually fail, and it's much more difficult to fix human bugs. Better to shoot for full automation.

heisnotanalien 11y ago

But a computer could do the same thing. It would be possible to alert on all the routes being blown away?

DanielDent 11y ago

Any sufficiently large system quickly reaches a point where a human has difficulty tracking what the system should look like. Google has at least tens of data center locations, each of which will have multiple physical failure domains. There are also many discontiguous routes being announced at all of their network PoPs. They have substantially more PoPs than data centers. It very quickly gets too much to reasonably expect people to be able to keep track of what the system should look like, let alone grasping what it does look like.

dsl 11y ago

First of all, BGP is core to Google's load balancing architecture. So within a single datacenter you probably have at least a few dozen devices down stream from each edge router. Secondly, I'm seeing just shy of 500 individual prefixes, 282 directly connected peers (other networks), and a presence at over 100 physical internet exchanges, just for one of Google's four ASes. Would you be able to read over that configuration and tell me if it has errors?

deleted 11y ago
[deleted]