4 ms·
>Our SRE team relies heavily on Glitter – our trusty SRE chatbot – to assist us with many of our operational tasks, so under normal circumstances, it’d be a sim
by l1n 9y ago
>Our SRE team relies heavily on Glitter – our trusty SRE chatbot – to assist us with many of our operational tasks, so under normal circumstances, it’d be a simple operation to disable these health checks in our Slack channel. However, things aren’t quite as straightforward as we’d expect, and the AWS outage has impacted Slack’s IRC gateway, which our chatbot relies on. This means we have to make all the necessary changes manually…
Interesting example of how infrastructure can become tangled in a way that makes crisis management much harder.
- subway 9y agoThis is an unfortunate example of "Ultron" style automation: https://queue.acm.org/detail.cfm?id=2841313 https://queue.acm.org/detail.cfm?id=2841313 Also kind of terrifying from a security perspective, since it's implied that Slack has been effectively delegated permission to modify AWS resources in their account.