3 ms·
Even with Kubernetes, you can clearly see what is deploying to what nodes. Not sure what Google's pipeline is, but I would suspect they have some "undo" functio
by sdan 7y ago
Even with Kubernetes, you can clearly see what is deploying to what nodes. Not sure what Google's pipeline is, but I would suspect they have some "undo" function to stop the deployment .
- reilly3000 7y agoI'm guessing it was something lower level than Kubernetes/Borg, since it was able to affect all of their networking bandwidth across multiple regions. ¯\_(ツ)_/¯
- shereadsthenews 7y agoThe interesting tidbit in here (really the only piece of information at all) is that the outage itself prevented remediation of the outage. That indicates that they might have somehow hosed their DSCP markings such that all traffic was undifferentiated. Large network operators typically implement a "network control" traffic class that trumps all others, for this reason.
- lclarkmichalek 7y agoWhen you're dealing with this scale of system, the number of config changes, automated or human, can make determining which config the harder issue. Once you've found out what the issue is, you probably also want the revert to go via the normal flows, for fear that your revert could exacerbate the situation. Both of those add time to remediation.