3 ms·
500 billions events. Always blows my mind how many people use aws
by jewba 1y ago
500 billions events. Always blows my mind how many people use aws
- Implicated 1y agoI know nothing. But I'd imagine the number of 'events' generated during this period of downtime will eclipse that number every minute.
- zimpenfish 1y ago"I felt a great disturbance in us-east-1, as if millions of outage events suddenly cried out in terror and were suddenly silenced" (Be interesting to see how many events currently going to DynamoDB are actually outage information.)
- nicce 1y agoI wonder how many companies have properly designed their clients. So that the timing before re-attempt is randomised and the re-attempt timing cycle is logarithmic.
- lan321 1y agoWhy randomized?
- adzm 1y agoHelps distribute retries rather than having millions synchronize
- yardstick 1y agoIt’s the Thundering Herd Problem. See https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem In short, if it’s all at the same schedule you’ll end up with surges of requests followed by lulls. You want that evened out to reduce stress on the server end.
- lan321 1y agoThank you. Bonsai and adzm as well. :)
- BonsaiAU 1y agoIt's just a safe pattern that's easy to implement. If your services back-off attempts happen to be synced, for whatever reason, even if they are backing off and not slamming AWS with retries, when it comes online they might slam your backend. It's also polite to external services but at the scale of something like AWS that's not a concern for most.
- jeffhuys 1y ago> they might slam your backend Heh
- 8note 1y agonowadays i think a single immediate retry is preferred over exponential backoff with jitter. if you ran into a problem that an instant retry cant fix, chances are you will be waiting so long that your own customer doesnt care anymore.
- mdavidn 1y agoMost companies will use the AWS SDK client's default retry policy.