4 ms·
Because there are unlimited number of failure modes for cascading failures. Going through all postmortems, you'll find it hard to find same kind of failures. Ev
by azurezyq 8y ago
Because there are unlimited number of failure modes for cascading failures. Going through all postmortems, you'll find it hard to find same kind of failures. Every time it's something unique.
Also it's difficult to get a good coverage by just pushing things to its limit. Let's say you have a system which has this RPC chain: A -> B -> C -> D. And not all requests cost the same and not all requests make way to D. In reality it's interesting that different composition of traffic may overload any of them. For example, some queries may hit expensive ML models while some other queries have a large number of candidates. In this case the former hits predictor service hard while the latter may hit index serving systems. I've actually read an outage happened during superbowl because sports-related queries are particularly expensive for some part of the system.