4 ms·
Two sentence summary: Slow clients strangled our service. The slow clients were caused by an external AWS networking issue.
by bscanlan 9y ago
Two sentence summary: Slow clients strangled our service. The slow clients were caused by an external AWS networking issue.
- rossdavidh 9y agoSeeing how the debugging process actually went, though, is also useful at times, to give you more examples of how other people work through problems where the cause is not clear.
- tetha 9y agoAnd it allows for reflection: How would our troubleshooting and alerting handle this? We had a similar haproxy session saturation problem a couple of month ago. By now, our alerting would pick this up within a minute and trigger alerts. Those include a runbook to resolve it. Our standard resolution would fail in a case like this, but I'm pretty sure we'd solve it in a second iteration.