4 ms·
Our incident management includes chatops automation that, among other things - generates a per-incident Google Doc - opens a per-incident Google Hangout - pa
by aaronharnly 8y ago
Our incident management includes chatops automation that, among other things
- generates a per-incident Google Doc
- opens a per-incident Google Hangout
- pages appropriate people via PagerDuty
During this incident (which affected our customers who use Google to log in), people had difficulty accessing all three (our PagerDuty uses Google Oauth).
This was a good reminder to have alternatives for all crucial incident management services. (We have backup plans in case Slack or our automation scripts are down, but didn’t have backups for Google...)
- shhehebehdh 8y agoGood advice. Within Google, they have backups that depend on no production infrastructure. Something as simple as an IRC server that is always running on prem can be invaluable.
- ViViDboarder 8y agoCan’t emphasize this enough! Outages will eventually happen, you need to plan for them.
- lathiat 8y agoOne of canonical’s employees was about to do a fairly important presentation (a few years ago) when our google oath went down. Could no longer load the slides from google docs and the had to make up new ones on the spot.