4 ms·
This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start
by cyberax 2mo ago
This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
- Kinrany 2mo agoI believe their point is that "system is at capacity" is something they ought to start fixing before the capacity is exceeded
- dcrazy 2mo agoBut then people like OP will claim that the capacity concerns are a lie manufactured to support an unjustified move to Azure.
- cyberax 2mo agoSure. But you might not even be realizing that something is just at the cusp if the load is spiky enough. The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing. AWS got a fair share of similar outages, so the newer SDKs now try to not exacerbate these kinds of issues: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr... The original AWS EBS outage is probably the canonical example: https://aws.amazon.com/message/65648/ https://aws.amazon.com/message/65648/