3 ms·
Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limi
by snewman 8y ago
Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point.
https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/ https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/
- adreamingsoul 8y agoThat's cool, but it's a bit more complicated then rate limits.
- codebje 8y agoDon't forget to alert when the double derivative of capacity growth is above a (very low!) threshold, that can catch an explosive problem far earlier than the 70% mark, by which time it might be growing at 5% per second.
- bostonvaulter2 8y agoThat sounds like a very interesting metric to track and report on. Do you have any further references that discuss that approach?
- codebje 8y agoNot offhand, sorry - I first encountered the notion at a conference some years back, but I've long since gotten out of operations so it's all a bit dusty for me now.
- cheeze 8y agoIt might also be a momentary blip that you really shouldn't need to wake a human up for. The hard part of monitoring imo is that perfect balance. Asymtotically approaching nirvana but never quite reaching.
- codebje 8y agoAll monitoring has that problem :-) But if your rate of rate of change has been positive for some suitable amount of time for the context, it's worth waking a human up over it, because the amount of resource remaining is dwindling exponentially.