3 ms·
I don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is
by tptacek 1y ago
I don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is some other technical change, maybe the issue is human/procedural. How could you possibly know? The owners of the system probably don't have a full answer within just a few hours of handling the incident!
- kqr 1y agoI would be worried if they had all the answers within a few hours! I was just caught off guard by the focus on technical control measures when there seems to have been fairly obvious problems with information channels. For example, "more global kill switches for features" is good, but would "only" have shaved 30 % off the time of recovery (if reading the timeline charitably). Being able to identify the broken component faster would have shaved 30–70 % off the time of recovery depending on how fast identification could happen – even with no improvements to the kill switch situation.