4 ms·
I just finished an 5 hour debugging session on what turned out to be several cascading bugs in one of the older systems at my company Can't deploy the fix, bec
by Scandiravian 6y ago
I just finished an 5 hour debugging session on what turned out to be several cascading bugs in one of the older systems at my company
Can't deploy the fix, because
- developers trigger deployments through slack and I don't have access to the underlying deployment system
- infrastructure guys who have access aren't responding to my emails
- vlod 6y agoSounds like a great opportunity to work on not having Slack as the only trigger mechanism. :) Or a least document how to call the slack bot manually. (assuming it's just a http endpoint)
- Scandiravian 6y agoI agree 100%! Though I think it might be dangerous to prepare for the "last disaster". It'll be some other system breaking next time, so I think we instead should identify what systems that do not have some kind of redundancy and determine blast radius of those crashing I'm good at not panicking about things I can't change, but I worry about some of my colleagues who find it difficult to not have control in these situations I can't do anything to help them at the moment, so for now I'm heading to my couch with my analogue book :)