3 ms·
No, TCP retries are not enough. TCP retries won't help if your backend is restarting, if failover switching is happening, if your overloaded cluster has just b
by nine_k 4y ago
No, TCP retries are not enough.
TCP retries won't help if your backend is restarting, if failover switching is happening, if your overloaded cluster has just been scaled up to add nodes, etc.
A reasonable application-level retry policy (exponential randomized delays, limited attempts) would turn these from a service disruption for the client into a mere delay, often pretty short.
- quadrifoliate 4y ago> TCP retries won't help if your backend is restarting, if failover switching is happening, if your overloaded cluster has just been scaled up to add nodes, etc. Yeah, this is where the nuance begins. You are correct that TCP is not always sufficient. Perhaps where we differ is that in my experience, it still helps for this to be a feature of the framework or infrastructure that the applications are running on (e.g. a retry budget in the service mesh, or a load balancer) rather than scattered around in the application itself. At some point it becomes a word game – you could say that the service mesh is also kind of an "application" itself, but the core principle is that the retries should be kept in a few simple, common places that are rarely tuned. Otherwise, you will find that N developers who are tasked with figuring out something like this will scatter N different version of your exponential randomized delay policy all across your codebase. It is always possible to avoid this with enough code review discipline, but once the trend starts, it's much harder to say "No, you need to fix this the right way".
- sarchertech 4y agoI’ve seen this exact problem many times. Counting TCP, the current system I’m working has 5 different layers of retries. TCP, the service mesh, the company wide http client, application retries, and a database backed queueing library that everyone uses. Imagine what happens when a service is flaky and each one of those layers retries 3-5 times.
- Jupe 4y agoYep. And then the user says.. "Oh, that didn't work - I'll just try again... and again..."
- foobiekr 4y agoIt’s pretty important to implement a mechanism to prevent a thundering herd if you can, and TCP doesn’t help here. Accidental synchronization can have many many causes. In addition, despite fantasies, tcp in particular is an instance level concept and reserving the ability to shoot instances without draining first is a good design. Application-level user-perceived reliability is an end to end concern, not a transport layer concern.