2 ms·
Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering. Source: decades of op
by stingraycharles 2mo ago
Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering.
Source: decades of operational pain.
- sqquima 2mo agoThis is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.
- MeetingsBrowser 2mo agoI think the “thundering herd” problem is pretty extensively documented
- stingraycharles 2mo agoThundering herds and circuit breakers are extremely well documented and practiced in production. AI absolutely knows about this. But knowing when to use which strategy and when a simple retry suffices is precisely the type of thing humans will remain to be better at than AI for the foreseeable future.
- __turbobrew__ 2mo agoDon't forget jitter as well. Otherwise your clients will just coordinate their retries.