15 ms·
The problem I had with implementing subscriptions with Stripe is that the batches of messages from the webhook could come back in clumps in random order. So the
by DonHopkins 8y ago
The problem I had with implementing subscriptions with Stripe is that the batches of messages from the webhook could come back in clumps in random order. So they would have references to objects that I hadn't been notified about their creation yet.
Theoretically and rarely a webhook could fail and be retransmitted arbitrarily later due to bad weather on the internet, so you have to be able to tolerate that, but practically and often it sent bunches of messages all at effectively the same time, which caused them to be processed by my web server in random order.
I finally gave up trying to structure the code so it could create objects in any order, and deal with objects it hadn't heard about yet, and just treated the webhook callback as a sign that I should soon make a request back to Stripe and ask them for ALL the events that had transpired.
So I'd log all the information in the webhook just for chuckles, then schedule a task that polled Stripe for batches of events, and dealt with them all at once without anything slipping through the cracks because of random reordering in transit.
- tnolet 8y agoHmm, interesting. Have not bumped into that one yet. This might be due to the fact I only use one webhook or that I'm a small time player. The volume of hooks is pretty managable right now.
- DonHopkins 8y agoGoing from straightforward one-time purchases to maintaining subscriptions for an online service is a huge jump in complexity! Then throw in free trials, prorating subscriptions, upgrading monthly to yearly, etc. There are so many issues and edge cases with recurring charges on credit cards that can blindside you.
- tnolet 8y agoFunny you say that, because I have actually had all of those cases occur and my current setup handled that just fine. Guess I'm lucky ;o)
- WestCoastJustin 8y agoI have run into this also using subscriptions + webhooks. For example, I am using the "charge.succeeded" webhook event to send a custom email receipt on my backend. But, for a few new customers there seems to be a race condition where the "charge.succeeded" webhook event will arrive, before the API call in my code returns a success, so there is now an event but I have no idea how to tie this back to a user. I am using the API to create a new customer and updating their user DB record with the customer token from stripe. So, I get into a situation where I do not know what customer to send the email too (because I don't have the customer token yet). I ended up just adding a HTTP 503 error (Service is Unavailable), for this specific webhook event when I cannot find the user, so the Stripe API will retry that event. This is a hack but it works. This just started happening a few weeks ago. There are a few of these things popping up here and there that I need to deal with but generally it works really well.
- DonHopkins 8y agoYeah, I think that's the one that got me. You don't actually want to do anything substantial in response to the webhook, anyway. Just log it to a queue and return immediately. Then some other process can come along and process the queue, taking as long as it needs to interrogate stripe, create users and objects in the database, and send email. At first that process was trying to build up my model piece by piece in response to each webhook callback, but they weren't being delivered in the right order all the time. Cache and log everything for auditing and debugging, but never use or depend on anything you have cached or that was delivered to you in a webhook event. Because of random ordering, it may already be stale by the time you get it. Always do a full refresh of all user and subscription data. In the end, that process got more robust and didn't actually give a shit about the particular events that were queued from the webhook. The fact that there was something in the queue just woke it up, then it figured out the transaction id of each event (which varied from event to event, but it ignored the most of the event data), and marked those transactions as needing to be refreshed. Then for each refreshable transaction, it would pull down fresh user and subscription data, and go from there. So it didn't matter what order events were pushed, or how many a flurry of events you got on the same transaction, because it processes all the affected transactions just once, instead of responding to each event.
- ThatPlayer 8y agoI did a subscription service with Stripe too several years ago. I think I just ended up failing those webhooks, and then letting Stripe automatically retry in a bit.
- GorgeRonde 8y agoThis is a problem of API async operations, either by nature or by implementation: for either reason the process at hand (payment whatever) relies on a webhook to continue. Here's a list of some problems it can bring: ## Synchronous context: - Network errors whose call can be retried. For whatever reason you know the message did not reach the remote API. Exponentially/randomly increase timeout between each call. Have multiple levels. Immediate retries vs retries delayed in a queue or stored for later inspection. - Network calls that fail and that you hesitate to retry. Typically timeouts. Was the query received on the other end of the tunnel ? You don't know. Ensure idempotency via an idempotency key or via a query to check if the entity has actually been created/updated. Both solutions require support from the API. This concerns both calls made by the client as well as those made by the API (to send notifications the the hooks). All of the above applies in addition to the following: ## Asynchronous context: ### Response/Webhook race condition: - Receiving the response as well as a 'created' event. You have to decide where to put what comes next after the API call in the process logic. Keep your code synchronous by placing the continuing code under the call. Or make it asynchronous by registering it to the event. - Receiving another event after the response. For instance, you make an API call, receive a 'created' event (ignored) followed by a 'successful' event (oops) before even receiving the response. You have a state to update, but can't find the right record because you haven't received the id from the response (and decided to ignore the 'created' event for the point above). ### Event ordering (quite the same thing as above): - Receiving events for the same entity in the wrong order. For instance: 'succeeded' before 'created'. - Receiving event for different entities in the wrong order. 'avatar uploaded' before 'account created'. ### Event number - None (add to that the absence of retries for notifications and you have a good part of maintenance cost of a system whose backend relies on web APIs). - Multiple. A typical solution is to store the received event with its id or any other way to ensure it is unique. What else ?
- ecnahc515 8y agoThis is my experience with most systems that send webhooks, in particular payments, and subscription management systems. As you've elaborated on in other comments, queuing and periodic retries are generally the best way to handle interacting with what is effectively, an eventually consistent API/system.