7 ms·

What I have always wondered about, in the stripe docs it says "Stripe's idempotency works by saving the resulting status code and body of the first request made

What I have always wondered about, in the stripe docs it says "Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeded or failed. Subsequent requests with the same key return the same result, including 500 errors." which indicates that the idempotency functionality is created in a kind of layer around the main application functionality, and thus the request from the idempotency layer to the main app is not itself idempotent. So the idempotency key protects against faults on the network between the client and stripe's idempotency layer, but not against stripe-internal faults between the idempotency layer and the application. Is that the case? Why is the idempotency not achieved by using an idempotent database transaction or atomic operation? What is the value of responding repeatedly with a 500 if the original 500 was caused by a transient error? Adyen seems to similarly implement the idempotency in a separate data store (https://docs.adyen.com/development-resources/api-idempotency#designing-for-resilience https://docs.adyen.com/development-resources/api-idempotency...). Both stripe's and adyen's implementation seem to not treat transient faults within their own systems correctly.

foolfoolz 5y ago
these seem extreme for most apis but are likely very good at preventing double charging. a “normal” api call usually costs nothing. i bet stripe and adyen have apis with an average cost per call (to someone) being like $20-$30 the cost to undo a charge is high in people and reputation
- polynomial 5y ago
  How could their API possibly cost $20-$30 per call? How could that even be a business model? Clearly, I am missing something here.
  - terinjokes 5y ago
    I suspect the OP meant to say charge instead of cost.
    - polynomial 5y ago
      ah that makes sense.
  - foolfoolz 5y ago
    unrelated but fun fact: AWS CloudHSM v1 (deprecated now) had a $5,000 api call. that was the cost to create a cluster.
  - wdb 5y ago
    Don’t some bank system transfer like the U.K. cost like ~£25 per transaction?
joshribakoff 5y ago
Many developers conflate idempotency and purity. It appears stripe is trying for purity, two requests always have the same response. This is likely misguided. Idempotency would be like the API failed to generate the success response, but the backend did issue the money movement. On subsequent retries what matters is that it doesn’t do a duplicate money movement. The client ought to get a request indicating the money was successfully moved (since it was), perhaps with the earlier timestamp as a way of indicating it was already moved. Or it ought to get a special response that indicates the client can stop retrying Sending the same error upon success sounds like the client has no way to know when to stop retrying. Sure it makes the API pure, but what problem does it even solve?
- rob-olmos 5y ago
  For Stripe, there's two headers that can help clients with retrying or not: "Idempotent-Replayed: true"[1] and "Stripe-Should-Retry"[2]. Whether they're helpful in all server-side error situations I'm not sure. 1: https://stripe.com/docs/idempotency#sending-idempotency-keys https://stripe.com/docs/idempotency#sending-idempotency-keys 2: https://stripe.com/docs/error-handling#the-stripe-should-retry-header https://stripe.com/docs/error-handling#the-stripe-should-ret...
  - nivertech 5y ago
    "Should-Retry" is useful, but "Idempotent-Replayed" while might be somewhat useful for debugging, makes the idempotent semantics worse: i.e. the side-effect idempotency stays the same, while result isn't idempotent. I guess in HTTP you may say that only the body of the HTTP response is the result, not the headers. But I implemented a similar system for GraphQL, and there are these boolean flags (isRetry and isReplayed, etc.) are part of the mutation inputs and/or payloads. The same with the "Idempotency-Key" itself, which is called clientMutationId in Relay mutation spec. The added value of not using HTTP headers is that it also works over websockets.
brandur 5y ago
> So the idempotency key protects against faults on the network between the client and stripe's idempotency layer, but not against stripe-internal faults between the idempotency layer and the application. Is that the case? Yes, in practice this is the case — the idempotency key insertion could succeed and then subsequent queries fail. Practically though, it doesn't happen very often — if a request gets as far as the idempotency layer successfully, the rest of it tends to work too. Where it doesn't, Stripe pays a lot of attention to 500s, especially where those pertain to charges, and a lot of time and energy is spent cleaning up state that might have resulted in an invalid transaction. > Why is the idempotency not achieved by using an idempotent database transaction or atomic operation? The most honest answer is that Stripe wasn't built on a data store where transactions are supported, so it wasn't even a possibility until quite recently, and by then the system was already well established in its current form. Beyond that though, once your requests are making their own requests to modify state in foreign systems (which is happening at Stripe in the form of banks, partners, internal systems, etc.), a single transaction isn't enough to keep things entirely in order anymore because it can't roll back that remote state. It is still possible to build a very robust system that is transaction-based, but it becomes a much more complex problem than a simple `BEGIN`/`COMMIT`.
- wonton53 5y ago
  > Practically though, it doesn't happen very often It probably happens much more often than you think if you say this as someone with an insider perspective. I have had to integrate with banking systems and payment systems that use this approach, and it is extremely frustrating and comes of as a way of offloading work to the client. If a payment capture succeedes but subsequently always returns 500 an api client has to first query the status and then execute if-then logic for something that could easily just have been a retry of the request (3x the logic). This is acceptable since there are a million other ways to mess up idem potency so integrations kind of end up this way anyway. But the worst part of such an approach is debugging the issues as a client. A client cannot see the internal logs and therefore have to call support which will ALWAYS answer «the transaction seem fine with us» and basically just close the case to protect their KPIs. Im pretty sure this type of issue has a name but cannot find it (state client see dont match the real state). I dont mean to offend the work people put into these APIs, but I cannot see the good qualities of this approach other than saving development hours (and possibly saving one db query, but as an effect you get a status request plus another capture request as a workaround from your clients).
donavanm 5y ago
I don’t speak for stripe, but do have similar experience with idempotent APIs. > the idempotency functionality is created in a kind of layer around the main application functionality … Yes. Think of it as a kind of request/response handler cache that doesn’t need to be deeply tied to the operation logic. On the request path insert an entry with a compound key for the “idempotency parameters” like client identity, operation, idempotency token, and request parameters. You can hash these if needed. On the response path update your idempotency cache with the response value. If you the underlying operation times out etc you can synthesize and insert the exception response. On subsequent requests read the idempotency key(s) from your cache and return the cached response if it exists. > So the idempotency key protects against faults on the network between the client and stripe's idempotency layer, but not against stripe-internal faults between the idempotency layer and the application. Maybe. The external idempotency token is there for client retries. Internally there’s idempotency all the way down to somewhere you’re (hopefully) doing a compare and swap, conditional put, or similar. A subsequent request that conflicts should be detected based on that state. > Why is the idempotency not achieved by using an idempotent database transaction or atomic operation? Beyond small examples you probably can’t. You end up calling multiple other services or have distributed state of some sort for most interesting operations. And distributed transactions are expensive/difficult/painful. So plumbing idempotency and error handling workflows through is simpler.

foolfoolz 5y ago

these seem extreme for most apis but are likely very good at preventing double charging. a “normal” api call usually costs nothing. i bet stripe and adyen have apis with an average cost per call (to someone) being like $20-$30 the cost to undo a charge is high in people and reputation

polynomial 5y ago
How could their API possibly cost $20-$30 per call? How could that even be a business model? Clearly, I am missing something here.
- terinjokes 5y ago
  I suspect the OP meant to say charge instead of cost.
  - polynomial 5y ago
    ah that makes sense.
- foolfoolz 5y ago
  unrelated but fun fact: AWS CloudHSM v1 (deprecated now) had a $5,000 api call. that was the cost to create a cluster.
- wdb 5y ago
  Don’t some bank system transfer like the U.K. cost like ~£25 per transaction?

joshribakoff 5y ago

Many developers conflate idempotency and purity. It appears stripe is trying for purity, two requests always have the same response. This is likely misguided. Idempotency would be like the API failed to generate the success response, but the backend did issue the money movement. On subsequent retries what matters is that it doesn’t do a duplicate money movement. The client ought to get a request indicating the money was successfully moved (since it was), perhaps with the earlier timestamp as a way of indicating it was already moved. Or it ought to get a special response that indicates the client can stop retrying Sending the same error upon success sounds like the client has no way to know when to stop retrying. Sure it makes the API pure, but what problem does it even solve?

rob-olmos 5y ago
For Stripe, there's two headers that can help clients with retrying or not: "Idempotent-Replayed: true"[1] and "Stripe-Should-Retry"[2]. Whether they're helpful in all server-side error situations I'm not sure. 1: https://stripe.com/docs/idempotency#sending-idempotency-keys https://stripe.com/docs/idempotency#sending-idempotency-keys 2: https://stripe.com/docs/error-handling#the-stripe-should-retry-header https://stripe.com/docs/error-handling#the-stripe-should-ret...
- nivertech 5y ago
  "Should-Retry" is useful, but "Idempotent-Replayed" while might be somewhat useful for debugging, makes the idempotent semantics worse: i.e. the side-effect idempotency stays the same, while result isn't idempotent. I guess in HTTP you may say that only the body of the HTTP response is the result, not the headers. But I implemented a similar system for GraphQL, and there are these boolean flags (isRetry and isReplayed, etc.) are part of the mutation inputs and/or payloads. The same with the "Idempotency-Key" itself, which is called clientMutationId in Relay mutation spec. The added value of not using HTTP headers is that it also works over websockets.

brandur 5y ago

> So the idempotency key protects against faults on the network between the client and stripe's idempotency layer, but not against stripe-internal faults between the idempotency layer and the application. Is that the case? Yes, in practice this is the case — the idempotency key insertion could succeed and then subsequent queries fail. Practically though, it doesn't happen very often — if a request gets as far as the idempotency layer successfully, the rest of it tends to work too. Where it doesn't, Stripe pays a lot of attention to 500s, especially where those pertain to charges, and a lot of time and energy is spent cleaning up state that might have resulted in an invalid transaction. > Why is the idempotency not achieved by using an idempotent database transaction or atomic operation? The most honest answer is that Stripe wasn't built on a data store where transactions are supported, so it wasn't even a possibility until quite recently, and by then the system was already well established in its current form. Beyond that though, once your requests are making their own requests to modify state in foreign systems (which is happening at Stripe in the form of banks, partners, internal systems, etc.), a single transaction isn't enough to keep things entirely in order anymore because it can't roll back that remote state. It is still possible to build a very robust system that is transaction-based, but it becomes a much more complex problem than a simple `BEGIN`/`COMMIT`.

wonton53 5y ago
> Practically though, it doesn't happen very often It probably happens much more often than you think if you say this as someone with an insider perspective. I have had to integrate with banking systems and payment systems that use this approach, and it is extremely frustrating and comes of as a way of offloading work to the client. If a payment capture succeedes but subsequently always returns 500 an api client has to first query the status and then execute if-then logic for something that could easily just have been a retry of the request (3x the logic). This is acceptable since there are a million other ways to mess up idem potency so integrations kind of end up this way anyway. But the worst part of such an approach is debugging the issues as a client. A client cannot see the internal logs and therefore have to call support which will ALWAYS answer «the transaction seem fine with us» and basically just close the case to protect their KPIs. Im pretty sure this type of issue has a name but cannot find it (state client see dont match the real state). I dont mean to offend the work people put into these APIs, but I cannot see the good qualities of this approach other than saving development hours (and possibly saving one db query, but as an effect you get a status request plus another capture request as a workaround from your clients).

donavanm 5y ago

I don’t speak for stripe, but do have similar experience with idempotent APIs. > the idempotency functionality is created in a kind of layer around the main application functionality … Yes. Think of it as a kind of request/response handler cache that doesn’t need to be deeply tied to the operation logic. On the request path insert an entry with a compound key for the “idempotency parameters” like client identity, operation, idempotency token, and request parameters. You can hash these if needed. On the response path update your idempotency cache with the response value. If you the underlying operation times out etc you can synthesize and insert the exception response. On subsequent requests read the idempotency key(s) from your cache and return the cached response if it exists. > So the idempotency key protects against faults on the network between the client and stripe's idempotency layer, but not against stripe-internal faults between the idempotency layer and the application. Maybe. The external idempotency token is there for client retries. Internally there’s idempotency all the way down to somewhere you’re (hopefully) doing a compare and swap, conditional put, or similar. A subsequent request that conflicts should be detected based on that state. > Why is the idempotency not achieved by using an idempotent database transaction or atomic operation? Beyond small examples you probably can’t. You end up calling multiple other services or have distributed state of some sort for most interesting operations. And distributed transactions are expensive/difficult/painful. So plumbing idempotency and error handling workflows through is simpler.