This totally happens, and is a thing, but tracing a "unexpected change in metrics on our DB server to "increase in metric showing call volume to DB server" to "

This totally happens, and is a thing, but tracing a "unexpected change in metrics on our DB server to "increase in metric showing call volume to DB server" to "increase in metric showing increase in fleet size of app servers that call the DB server" takes minutes to identify, not 24 hours. And the remediation procedure is quite simple too - scale down. If the fleet can't handle it, increase your throttling. If even with increased throttling you can't handle the load and everybody's timing out (but you still can't scale out), you start spinning up new load balancers, and applying weights in DNS where at least some percentage of your callers are getting through while others see full failure. The key point is in anything related to such an outage, even if it takes a long time to recover, your customers shouldn't see a full outage. So it was either an entirely different class of problems or Roblex has a huge ops skills gap. I'm betting the former. edit: never mind, maybe it's the latter after all: https://twitter.com/NIDeveloper/status/1454773313792880640 https://twitter.com/NIDeveloper/status/1454773313792880640