This sort of architecture often shows up in actors, e.g. https://www.microsoft.com/en-us/research/publication/orleans-distributed-virtual-actors-for-programmabi

This sort of architecture often shows up in actors, e.g. https://www.microsoft.com/en-us/research/publication/orleans-distributed-virtual-actors-for-programmability-and-scalability/ https://www.microsoft.com/en-us/research/publication/orleans... In that world, what you're generally looking at is local state + incoming message -> new state + outgoing messages Outgoing messages are sent only after persisting, and will be retried until success. Unique message IDs are used for idempotency (also for incoming messages). Important: the actor runs with no side effects -- that's what makes rerunning things safe. In that kind of world, side effects are often achieved via e.g. sending a message that updates a materialized view somewhere (see e.g. Kafka). With this setup, the source of badness is often isolated to the incoming message, and after a few failures an incoming message can be moved into a "dead letter queue" for ops to look at. In many scenarios, this actually works remarkably well. https://pmatseykanets.github.io/beanstalkd-docs/protocol/#bury https://pmatseykanets.github.io/beanstalkd-docs/protocol/#bu...