3 ms·

> This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams, which impaired their ability to find the

• femiagbabiaka 5y ago
  It is. And to add, all automation that we rely on in peace time can often complicate cross cutting wartime incidents by raising the ambient complexity of an environment. Bainbridge for more: https://blog.acolyer.org/2020/01/08/ironies-of-automation/ https://blog.acolyer.org/2020/01/08/ironies-of-automation/
• daenney 5y ago
  Yup. There was a GCP outage a couple of years ago like this. I don’t remember the exact details, but it was something along the lines of a config change went out that caused systems to incorrectly assume there were huge bandwidth constraints. Load shedding kicked in to drop lower priority traffic which ironically included monitoring data rendering GCP responders blind and causing StackDriver to go blank for customers.
• sponaugle 5y ago
  Indeed - Even the recent facebook outage outlined how slow recovery can be if the primary investigation and recovery methods are directly impacted as well. Back in the old days some environments would have POTS dial-in connections to the consoles as backup for network problems. That of course doesn't scale, but it was an attempt to have an alternate path of getting to things. Regrettably if a backhoe takes out all of the telecom at once that plan doesn't work so well.