3 ms·
Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times
by taylor-s 3mo ago
Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.
- Quarrelsome 3mo agoisn't that a bomb with a pair of scissors to cut the fuse that could break down under certain conditions? I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?
- stingraycharles 3mo agoGoogle “thundering herd” and you’ll understand why uncontrolled retries can be / are bad.
- inigyou 3mo agoQuarrelsome is suggesting you should have no retries, not unlimited.
- stingraycharles 3mo agoYes I’m aware, and they’re probably suggesting that something on a higher level should instead retry, and I’m arguing that you need to coordinate the retry mechanisms and behavior between differently layers of your stack.
- MeetingsBrowser 3mo agoExponential back off retries could hide a real issue, but I would estimate something like 99.99999999% of network retries are resolved within the first 2 attempts. Not using retries is optimizing for the astronomically rare case, which is better mitigated by other means
- stingraycharles 3mo agoRetries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering. Source: decades of operational pain.
- sqquima 3mo agoThis is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.
- MeetingsBrowser 3mo agoI think the “thundering herd” problem is pretty extensively documented
- stingraycharles 3mo agoThundering herds and circuit breakers are extremely well documented and practiced in production. AI absolutely knows about this. But knowing when to use which strategy and when a simple retry suffices is precisely the type of thing humans will remain to be better at than AI for the foreseeable future.
- __turbobrew__ 3mo agoDon't forget jitter as well. Otherwise your clients will just coordinate their retries.