> Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...] > I disagree I too disagree with that too, pretty strongly. The network is unr

> Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...] > I disagree I too disagree with that too, pretty strongly. The network is unreliable. Technology has become pretty good, to the point that we have become spoiled and take the reliability for granted. But it is at its very core an unreliable medium. Applications should expect that and survive accordingly. Now, before packets go out to the network, or after they have made their way inside your network, they are in an environment that you have a degree of control. Errors and retries should be monitored. If they increase and remain elevated, they should be investigated. But guess what, if your services are resilient, you should have some time to investigate before things start breaking down - as they would if you treated everything as if it was _localhost_ Case in point: our app started breaking in horrendous ways once we deployed it in China and it tried to cross over the Great Firewall. Most issues would have been survivable if they just backed off and retried. The GFW doesn't usually like the first attempt to/from a new address, but it will usually allow the traffic after a while. We had to pay a company a not insignificant chunk of cash to get better connectivity (that would have been fine as an improvement, but not to make the system work at all). Retries are at the core of everything we do. TCP has retries (if the author followed their own advice, they should switch to UDP!). Kubernetes has a whole bunch of retries before reaching error states: CrashLoopBackOff, ImagePullBackOff. Your ethernet card has retries. But it also tracks errors. Track errors. And have retries. For as long as it makes sense to retry for your use-case. > well then perhaps implement better logging Not logging. Metrics! Logs are good for troubleshooting. For signaling issues, metrics are better. Errors can be a simple counter that you increment. Have Prometheus or similar scrape that and throw alerts as necessary.