This is a problem of API async operations, either by nature or by implementation: for either reason the process at hand (payment whatever) relies on a webhook t

This is a problem of API async operations, either by nature or by implementation: for either reason the process at hand (payment whatever) relies on a webhook to continue. Here's a list of some problems it can bring: ## Synchronous context: - Network errors whose call can be retried. For whatever reason you know the message did not reach the remote API. Exponentially/randomly increase timeout between each call. Have multiple levels. Immediate retries vs retries delayed in a queue or stored for later inspection. - Network calls that fail and that you hesitate to retry. Typically timeouts. Was the query received on the other end of the tunnel ? You don't know. Ensure idempotency via an idempotency key or via a query to check if the entity has actually been created/updated. Both solutions require support from the API. This concerns both calls made by the client as well as those made by the API (to send notifications the the hooks). All of the above applies in addition to the following: ## Asynchronous context: ### Response/Webhook race condition: - Receiving the response as well as a 'created' event. You have to decide where to put what comes next after the API call in the process logic. Keep your code synchronous by placing the continuing code under the call. Or make it asynchronous by registering it to the event. - Receiving another event after the response. For instance, you make an API call, receive a 'created' event (ignored) followed by a 'successful' event (oops) before even receiving the response. You have a state to update, but can't find the right record because you haven't received the id from the response (and decided to ignore the 'created' event for the point above). ### Event ordering (quite the same thing as above): - Receiving events for the same entity in the wrong order. For instance: 'succeeded' before 'created'. - Receiving event for different entities in the wrong order. 'avatar uploaded' before 'account created'. ### Event number - None (add to that the absence of retries for notifications and you have a good part of maintenance cost of a system whose backend relies on web APIs). - Multiple. A typical solution is to store the received event with its id or any other way to ensure it is unique. What else ?