I found myself wishing for a few code snippets here. It would be interesting. A lot of time code that handles "connection refused" or fast failures doesn't hand

I found myself wishing for a few code snippets here. It would be interesting. A lot of time code that handles "connection refused" or fast failures doesn't handle network slowness well. I've seen outages from "best effort" services (and the best-effort-ness worked when the services were hard down) because all of a sudden calls that were taking 50 ms were not failing but all taking 1500+ ms. Best effort but no client enforced SLAs that were low enough to matter. Load shedding never kicked in, so things had to be shutdown for a bit and then restarted. Seems their normal operating state might be what is called "meta-stable" - dynamically stable at a high thru-put (edited) unless/until a brief glitch bumps the system into the low work being finished state which is also stable.