I've found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades

I've found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades or maintenance. Common example is data being pushed between systems with HTTP. Take the simplest case of propagating a boolean value. You toggle some setting in the UI, and it sends an update to another system with an HTTP request, retrying on a delay if it can't connect. This has two problems. The first is that if the user toggles a setting on and then off, you can have two sets of retries going, producing a random result when the far end can be connected to again. The second, is that the machine doing the retries might get rebooted, and people often fail to persist the fact that a change needs to be pushed to the other system. I've seen this issue between two processes on the same machine, so technically you don't even need a network.