In my work in severless, we see 10-40 (known) dropped requests power day. I highlight known because many requests fail with transient network errors that never

In my work in severless, we see 10-40 (known) dropped requests power day. I highlight known because many requests fail with transient network errors that never make it to our logs, and others fail somewhere in the cloud provider black box before we get a log, but the caller gets a response from the cloud providers load balancer. While 10-40 is not a lot in the scale of hundreds of thousands of requests per day (were not that large), when that request is a write event, that's very detrimental to customer perception/experience... It might even cost your business money (failing to turn off/delete something)... And the easy answer of "retry the request" fails with write events that are not idempotent... WHEN should you retry? A gateway error doesn't necessarily mean the back end didn't get the request.. just that your didn't get a response. It's even worse when what you'd expect to be internal communication has this same failure rate.. your back end processes getting a random 504 gateway error in places it never happened before.. breaking longer running processes.