3 ms·

I have run into this also using subscriptions + webhooks. For example, I am using the "charge.succeeded" webhook event to send a custom email receipt on my back

I have run into this also using subscriptions + webhooks. For example, I am using the "charge.succeeded" webhook event to send a custom email receipt on my backend. But, for a few new customers there seems to be a race condition where the "charge.succeeded" webhook event will arrive, before the API call in my code returns a success, so there is now an event but I have no idea how to tie this back to a user. I am using the API to create a new customer and updating their user DB record with the customer token from stripe. So, I get into a situation where I do not know what customer to send the email too (because I don't have the customer token yet). I ended up just adding a HTTP 503 error (Service is Unavailable), for this specific webhook event when I cannot find the user, so the Stripe API will retry that event. This is a hack but it works. This just started happening a few weeks ago. There are a few of these things popping up here and there that I need to deal with but generally it works really well.

DonHopkins 8y ago
Yeah, I think that's the one that got me. You don't actually want to do anything substantial in response to the webhook, anyway. Just log it to a queue and return immediately. Then some other process can come along and process the queue, taking as long as it needs to interrogate stripe, create users and objects in the database, and send email. At first that process was trying to build up my model piece by piece in response to each webhook callback, but they weren't being delivered in the right order all the time. Cache and log everything for auditing and debugging, but never use or depend on anything you have cached or that was delivered to you in a webhook event. Because of random ordering, it may already be stale by the time you get it. Always do a full refresh of all user and subscription data. In the end, that process got more robust and didn't actually give a shit about the particular events that were queued from the webhook. The fact that there was something in the queue just woke it up, then it figured out the transaction id of each event (which varied from event to event, but it ignored the most of the event data), and marked those transactions as needing to be refreshed. Then for each refreshable transaction, it would pull down fresh user and subscription data, and go from there. So it didn't matter what order events were pushed, or how many a flurry of events you got on the same transaction, because it processes all the affected transactions just once, instead of responding to each event.
danpalmer 8y ago
We just use a database lock for this. Lock on the charge/order/user/whatever, then the stripe webhook delivery is blocked until the transaction commits. This might seem like a pain, but Stripe have no idea how fast their clients are, there’s not much point in waiting, say, 1 second minimum before delivering the webhook, as that only alleviates the problem for clients fast enough to “finish” within 1 second, whatever their definition of finish is. For this reason, I think Stripe are doing the right thing, and I’m not sure there’s much they can do to make it much easier. Once you know it needs to be done, a solution is pretty straightforward.
- DonHopkins 8y ago
  Yikes! You really should respond to the webhook handler immediately, by queuing it, and handling it asynchronously in another process. It's bad enough to perform slow operations like creating lots of objects in the database, calling other web apis, sending email, before you return a success to stripe's webhook. But for one webhook to lock out all other webhooks while it did all that work would only compound the problem. Stripe tends to send a whole flurry of events related to the same transaction, when there's actually only one thing for you to do in one swoop (create a user and start their subscription), so to processes them one by one is very inefficient, especially if you have to call back to stripe for each event.
  - ct520 8y ago
    these post makes me cringe as someone that deals with enterprise sass integrations that publish to webhooks. Lot of bad actors seen many interesting issues pop up because the awesome stuff subscribers try to do
  - danpalmer 8y ago
    Sorry, maybe should have clarified, webhooks are typically processed in a queue, we reply to Stripe immediately, but we block the processing of the webhook on that lock. As we're processing in a queue with multiple workers this typically doesn't block much work from happening. There are some cases where we specifically want to propagate the error to a webhook provider (not in the case of Stripe, so we work inline, but that's rare). There are also some cases where we want to process webhooks on a serial queue, one at a time, to ensure in-order delivery (again not in the case of Stripe).
ajford 8y ago
Gotta say, that doesn't sound like a hack, it sounds like a perfectly normal response to the imperfect nature of async processes and the internet itself. TCP traffic itself behaves in the same way. If a packet arrives out of order, you discard it and notify the other end that the packet was not delivered, allowing it to retry. This is just the same behavior. Web services at their finest.

DonHopkins 8y ago

Yeah, I think that's the one that got me. You don't actually want to do anything substantial in response to the webhook, anyway. Just log it to a queue and return immediately. Then some other process can come along and process the queue, taking as long as it needs to interrogate stripe, create users and objects in the database, and send email. At first that process was trying to build up my model piece by piece in response to each webhook callback, but they weren't being delivered in the right order all the time. Cache and log everything for auditing and debugging, but never use or depend on anything you have cached or that was delivered to you in a webhook event. Because of random ordering, it may already be stale by the time you get it. Always do a full refresh of all user and subscription data. In the end, that process got more robust and didn't actually give a shit about the particular events that were queued from the webhook. The fact that there was something in the queue just woke it up, then it figured out the transaction id of each event (which varied from event to event, but it ignored the most of the event data), and marked those transactions as needing to be refreshed. Then for each refreshable transaction, it would pull down fresh user and subscription data, and go from there. So it didn't matter what order events were pushed, or how many a flurry of events you got on the same transaction, because it processes all the affected transactions just once, instead of responding to each event.

danpalmer 8y ago

We just use a database lock for this. Lock on the charge/order/user/whatever, then the stripe webhook delivery is blocked until the transaction commits. This might seem like a pain, but Stripe have no idea how fast their clients are, there’s not much point in waiting, say, 1 second minimum before delivering the webhook, as that only alleviates the problem for clients fast enough to “finish” within 1 second, whatever their definition of finish is. For this reason, I think Stripe are doing the right thing, and I’m not sure there’s much they can do to make it much easier. Once you know it needs to be done, a solution is pretty straightforward.

DonHopkins 8y ago
Yikes! You really should respond to the webhook handler immediately, by queuing it, and handling it asynchronously in another process. It's bad enough to perform slow operations like creating lots of objects in the database, calling other web apis, sending email, before you return a success to stripe's webhook. But for one webhook to lock out all other webhooks while it did all that work would only compound the problem. Stripe tends to send a whole flurry of events related to the same transaction, when there's actually only one thing for you to do in one swoop (create a user and start their subscription), so to processes them one by one is very inefficient, especially if you have to call back to stripe for each event.
- ct520 8y ago
  these post makes me cringe as someone that deals with enterprise sass integrations that publish to webhooks. Lot of bad actors seen many interesting issues pop up because the awesome stuff subscribers try to do
- danpalmer 8y ago
  Sorry, maybe should have clarified, webhooks are typically processed in a queue, we reply to Stripe immediately, but we block the processing of the webhook on that lock. As we're processing in a queue with multiple workers this typically doesn't block much work from happening. There are some cases where we specifically want to propagate the error to a webhook provider (not in the case of Stripe, so we work inline, but that's rare). There are also some cases where we want to process webhooks on a serial queue, one at a time, to ensure in-order delivery (again not in the case of Stripe).

ajford 8y ago

Gotta say, that doesn't sound like a hack, it sounds like a perfectly normal response to the imperfect nature of async processes and the internet itself. TCP traffic itself behaves in the same way. If a packet arrives out of order, you discard it and notify the other end that the packet was not delivered, allowing it to retry. This is just the same behavior. Web services at their finest.