4 ms·
This is why you always use exponential backoff.
by brigadier132 3y ago
This is why you always use exponential backoff.
- fathyb 3y agoAnd when you're at Twitter scale, sprinkle some jitter too.
- oblio 3y agoWhat do you mean?
- jyxent 3y agoAdding some randomization to the exponential backoff times to avoid the thundering herd problem: https://en.wikipedia.org/wiki/Thundering_herd_problem https://en.wikipedia.org/wiki/Thundering_herd_problem
- wolfgang42 3y agoSay you have a bug that caused 100,000 HTTP requests to hang, and you kick the node and make them all fail at once. One second later, 100,000 clients suddenly retry simultaneously, causing a huge spike in load which makes most of their requests fail. They use exponential backoff, so two seconds after that, 99,000 clients retry, causing a huge spike in load that makes most of their requests fail. Four seconds after that, 98,000 clients retry... If you introduce a bit of randomness into the retry timing (say, multiply by 1.8~2.2 instead of a straight doubling), that thundering herd will spread itself out and be much easier to recover from.
- 8organicbits 3y agoJitter is a little randomness in how long clients wait between retries. It ensures that you don't have a "thundering herd" all retrying at the same time. Imagine if your API used exponential backoff of [1s, 2s, 4s, 8s, ...] and a large group of requests gets a retryable error at t=0. They will all retry at exactly t=1, t=2, etc. If the group is large enough that repeated surge of requests can knock you offline. There's nasty form of this where the site is offline for a bit and then all the clients rush their requests in when it comes back online. The client requests are all coordinated on the site recovery time and end up overloading the site with their coordinated retries.
- henry2023 3y agoI enjoyed your comment a lot. Interesting to think that abstract structures like a load balanced webserver have a simil to the fundamental frequency observer in physical structure
- nightshadetrie 3y agoAlso useful in caching mechanisms.
- doomleika 3y agoAh, so it’s like CDMA in WLAN, TIL
- bezout 3y agoTIL about jitter
- globalise83 3y agoThis is why you SHOULD always use exponential backoff. ;)
- mmastrac 3y agoThere may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. (thanks RFC 2119)
- stan_kirdey 3y agoself ddos with backoff, :chef kiss:
- Waterluvian 3y agoI wonder if exponential backoff should be the default behaviour for request libraries/APIs. Their default of “just go ham on that API” feels like the same footgun of “by default this Humongous Database is wide open.”
- DropInIn 3y agoI doubt it ever would become the standard unless everyone was using third party libraries that forced it in some way, most likely opaque by default which would cause plenty of devs headaches, right? The easiest path will always be the default for the majority of devs, with a simple "timer" type solution being the easiest to implement in pretty much all cases except where otherwise it's literally forced on them.