4 ms·

Backpressure is definitely important--and I have even been largely sold over the past few years on the final point about queues of any size potentially being ha

by saurik 6y ago

Backpressure is definitely important--and I have even been largely sold over the past few years on the final point about queues of any size potentially being harmful--but this article failed to motivate it correctly by framing the problem in terms of a system that had an even more common (and dire) mistake: the existence of any form of automated timeout, which led requests that would eventually have worked to get retried... quite possibly now as a duplicate task; the mere removal of the timeout would probably make this work correctly, as there certainly will be some "back pressure of last resort" somewhere (such a maximum number of pending database transactions due to some maximum number of connections to the server due to some maximum number of socket descriptors or ephemeral ports or something). You simply can't predict how long something might take to happen, so you shouldn't. You shouldn't have a "default" timeout of 10 seconds "because of course it should complete by then". You don't know my networking connection and you don't know your congestion and load: the number of times I have had to fix something broken by some inane timeout--the most recent being a bug I found in the default Google Play Store Python library HTTP client setup that kept timing out my giant APK uploads that were even making progress--is ridiculous: you should have to call together everyone at your company to get consensus and then bring in a second external opinion before you even consider adding a timeout (and like, there are places where timeouts might make sense, but they have to be so so carefully done; a SYN packet handshake timeout is probably one of these places).

wahern 6y ago
IME, people add these timeouts as a hack around two major things: 1) broken software (often their own) that can lock up 2) broken networks that prevent proper dead peer detection, such as by dropping ICMP packets, employing NAT or other stateful firewall rules that drop unknown packets (which happens when a router reboots or otherwise loses or evicts state), etc. The eternal irony is that these intended robustness hacks invariably end up creating or compounding reliability problems. And people get ridiculously sophisticated with it. For example, rather than add timeouts to each piece of client software, people may an employ a forwarding application proxy (see, e.g., istio on kubernetes) to front requests and implement retries centrally. Now instead of N problems you have 2N or N^2 problems because such architectures invariably end up recapitulating all the sins committed at the IP and TCP level.

by saurik 6y ago

wahern 6y ago

IME, people add these timeouts as a hack around two major things: 1) broken software (often their own) that can lock up 2) broken networks that prevent proper dead peer detection, such as by dropping ICMP packets, employing NAT or other stateful firewall rules that drop unknown packets (which happens when a router reboots or otherwise loses or evicts state), etc. The eternal irony is that these intended robustness hacks invariably end up creating or compounding reliability problems. And people get ridiculously sophisticated with it. For example, rather than add timeouts to each piece of client software, people may an employ a forwarding application proxy (see, e.g., istio on kubernetes) to front requests and implement retries centrally. Now instead of N problems you have 2N or N^2 problems because such architectures invariably end up recapitulating all the sins committed at the IP and TCP level.