6 ms·

This is all way too much. If you see a duplicate idempotency key, skip the replay and always return 409. This becomes a client problem. Clients already need to

This is all way too much. If you see a duplicate idempotency key, skip the replay and always return 409. This becomes a client problem. Clients already need to help enforce idempotent contracts; "check for conflict response" is not an onerous imposition. I've built multiple ecommerce APIs with this approach and they work great. No heroic measures required. You can often satisfy this contract with a unique constraint; if not, a simple presence check in redis. No hashing or worrying about PII. My rant about this: https://github.com/stickfigure/blog/wiki/How-to-%28and-how-not-to%29-design-REST-APIs#rule-11-do-provide-idempotence-mechanisms https://github.com/stickfigure/blog/wiki/How-to-%28and-how-n...

Pxtl 6mo ago
[dead]
halestock 6mo ago
But that's not idempotent? If I'm a client and I don't know if the original request went through, getting a 409 on any subsequent requests tells me nothing about whether the original request was successful or not.
- onionisafruit 6mo ago
  The 409 should come with an id or a url the client can use to find the original result.
  - stickfigure 6mo ago
    ...and if you're using the approach of "let the client pick ids", you don't even need that. The client has everything it needs.
    - deleted 5mo ago
      [deleted]
- stickfigure 6mo ago
  Retries will only receive 409 if the original request was successful. If the original request failed, the server performs the operation as normal on the second request. It doesn't replay failures. The whole point of the idempotence mechanism is so you can make a reliable distributed system. If the first try fails, the client doesn't know if it succeeded or not, so the client should try again later ("at-least-once"). The idempotence mechanism just ensures that we don't get duplicates in the case that the first try actually succeeded. If you replayed failures there wouldn't be any point to the idempotency key.
  - pdonis 5mo ago
    What if the original request is still being processed when the retry comes in? That doesn't fall into either of your categories: the request isn't successful, but it hasn't failed either.
    - stickfigure 5mo ago
      That's usually solved with traditional database transactions. Even if you have a complex long-running multistep orchestration problem, you can break it down into simpler transactions. Eg you could start with a "lock the resources" txn. But 99% of these conversations around idempotence are simple POST operations like "create order" that regular old database concurrency management handles just fine.
      - pdonis 5mo ago
        > That's usually solved with traditional database transactions. That doesn't answer my question. What response do you return to the client in the case I described?
        matsemann 5mo ago
        The lock would normally make the second request wait, aka not return a response, until the first one is done. Then it sees it's a duplicate and returns that. Or it times out and returns an error. Then the client hopefully have some exponential back off strategy, so the third attempt doesn't suffer the same fate.
- hamdingers 5mo ago
  If you're a client using the same idempotency key for a materially different request you have a bug.
  - theptip 5mo ago
    Yes, and if you are building a payment API you need to be robust to client bugs.
    - anamexis 5mo ago
      Rejecting the conflicting request is being robust to improperly reused idempotency keys. There's no other reasonable answer.
    - jeremyjh 5mo ago
      The client is part of a distributed transaction. It can't be oblivious to this. Clear semantics and accurate adherence to them is the only answer that doesn't make the overall system unsound. Client bugs are expected and so the simplest semantics that ensure data integrity and accurate responses are the best way to help them identify and fix their bugs.
- kelnos 5mo ago
  You provide in the response body a JSON blob or something that indicates a detailed error code, like DUPLICATE_IDEMPOTENCY_KEY, with some more information that points to the record ID of the corresponding transaction that the client can fetch. Sure, a strict reading of "idempotence" might require that the response for subsequent requests be identical to the first, but for practical concerns, what matters is the API contract you define, document, and adhere to. The purpose of idempotence is to ensure that you don't end up with duplicate transactions. That's what actually matters. How that's represented in the protocol is an implementation detail.
pverheggen 5mo ago
I really like this approach, and am going to keep it in my back pocket. However, it's good for e-commerce where there are a subset of "important" operations, but I'd argue the idempotency key is better for a financial API like Stripe where the majority of your operations need these semantics. You also run into a problem if PUT/PATCH needs to be exactly-once instead of at least once. Not as common, but again, might be something you run into with a financial API.
- stickfigure 5mo ago
  There's nothing special about any particular HTTP method; it is the inherent nature of distributed systems that you can not have exactly-once semantics with a single HTTP request. To create exactly-once semantics, you need a client which retries until success and a server which prevents duplicates. The only difference between the approach I'm describing and Stripe's approach is the detail of how the client knows that it's done. "My" mechanism (notify the client that a request is a duplicate) was dominant in financial transaction processing systems before Stripe; I used probably a half dozen of them back in the day. Stripe came along and from the beginning decided "we want to compete based on making a friendly API". They succeeded, and if you ever look at Paypal's original API, it's easy to see why. It's true that repeating successful API responses makes life slightly easier for clients; they never need to check for a 409 (or whatever) response. It comes at a cost of making life quite a bit more complex on servers. Personally I don't think the tradeoff is worth it, but YMMV. If your single competitive advantage is "easy API" maybe it makes sense. If you're normal B2B, almost certainly not.
  - saltcured 5mo ago
    FWIW, I enjoyed your engagement throughout the thread above, fighting the good fight. I think there is also a risk of an "easy API" when it leads to magical thinking and sloppy development. If the naive client programmer starts to think the reliability is handled for them, they may also flub the handling of the idempotence key that remains the crux here. E.g. not persisting them well enough and returning to a situation where the user can accidentally make duplicate payments just like the naive system with no idempotence feature...
mamcx 5mo ago
And a good way to convey this contract is force the clients to hash-chain the calls, so is now more clear that it should do (and even can detect the send will fail at client side)
deleted 5mo ago
[deleted]
deleted 5mo ago
[deleted]
jeremyjh 5mo ago
I really like that article, thank you for sharing it again. I wish I had read it a decade ago, or even the first time you submitted it - I had to learn some of these things the hard way. I agree with you - I don't think Stripe has made the right choices here and its unfortunate that it has inspired so many other people to make the same poor choices. I don't agree that their system is as sound as always returning 409s. I think having a short window where you return response bodies is fine, but after that they should still be sending 409s. If no one will ever actually resend a request after 24 hours how is it not fine to send 409s when they do? They've chosen to implement the more expensive choice and then not back it with the cheapest one.
- deleted 5mo ago
  [deleted]
reactordev 5mo ago
This is because there are countless tutorials and slop on the internet that says, no, instructs, you to handle idempotency on the client instead of where it belongs. Server-side.
malux85 5mo ago
Exactly - junior engineers haven't yet developed a good taste of "how much the system should try and deal with errors" - so they always go way overboard to guess and try and fix errors way too much. I agree with you - this is a client error - 409 and make them fix the bug on their side. Clean, reliable, simple, separation of responsibility.
kgwxd 5mo ago
> If you see a duplicate idempotency key I'm no expert but an "idempotency key" already has some major smell to it.
- bcrosby95 5mo ago
  Why? I've used plenty of systems that have an idempotency key. E.g. many payment processors will take an order id and won't let you charge an order id more than once. That's just an idempotency key by another name.
- yencabulator 5mo ago
  Then learn from the experts, maybe? https://docs.stripe.com/api/idempotent_requests https://docs.stripe.com/api/idempotent_requests
fuzzythinker 5mo ago
Thank you, like and agree with the rest of the rules

Pxtl 6mo ago

[dead]

halestock 6mo ago

But that's not idempotent? If I'm a client and I don't know if the original request went through, getting a 409 on any subsequent requests tells me nothing about whether the original request was successful or not.

onionisafruit 6mo ago
The 409 should come with an id or a url the client can use to find the original result.
- stickfigure 6mo ago
  ...and if you're using the approach of "let the client pick ids", you don't even need that. The client has everything it needs.
  - deleted 5mo ago
    [deleted]
stickfigure 6mo ago
Retries will only receive 409 if the original request was successful. If the original request failed, the server performs the operation as normal on the second request. It doesn't replay failures. The whole point of the idempotence mechanism is so you can make a reliable distributed system. If the first try fails, the client doesn't know if it succeeded or not, so the client should try again later ("at-least-once"). The idempotence mechanism just ensures that we don't get duplicates in the case that the first try actually succeeded. If you replayed failures there wouldn't be any point to the idempotency key.
- pdonis 5mo ago
  What if the original request is still being processed when the retry comes in? That doesn't fall into either of your categories: the request isn't successful, but it hasn't failed either.
  - stickfigure 5mo ago
    That's usually solved with traditional database transactions. Even if you have a complex long-running multistep orchestration problem, you can break it down into simpler transactions. Eg you could start with a "lock the resources" txn. But 99% of these conversations around idempotence are simple POST operations like "create order" that regular old database concurrency management handles just fine.
    - pdonis 5mo ago
      > That's usually solved with traditional database transactions. That doesn't answer my question. What response do you return to the client in the case I described?
      - matsemann 5mo ago
        The lock would normally make the second request wait, aka not return a response, until the first one is done. Then it sees it's a duplicate and returns that. Or it times out and returns an error. Then the client hopefully have some exponential back off strategy, so the third attempt doesn't suffer the same fate.
hamdingers 5mo ago
If you're a client using the same idempotency key for a materially different request you have a bug.
- theptip 5mo ago
  Yes, and if you are building a payment API you need to be robust to client bugs.
  - anamexis 5mo ago
    Rejecting the conflicting request is being robust to improperly reused idempotency keys. There's no other reasonable answer.
  - jeremyjh 5mo ago
    The client is part of a distributed transaction. It can't be oblivious to this. Clear semantics and accurate adherence to them is the only answer that doesn't make the overall system unsound. Client bugs are expected and so the simplest semantics that ensure data integrity and accurate responses are the best way to help them identify and fix their bugs.
kelnos 5mo ago
You provide in the response body a JSON blob or something that indicates a detailed error code, like DUPLICATE_IDEMPOTENCY_KEY, with some more information that points to the record ID of the corresponding transaction that the client can fetch. Sure, a strict reading of "idempotence" might require that the response for subsequent requests be identical to the first, but for practical concerns, what matters is the API contract you define, document, and adhere to. The purpose of idempotence is to ensure that you don't end up with duplicate transactions. That's what actually matters. How that's represented in the protocol is an implementation detail.

pverheggen 5mo ago

I really like this approach, and am going to keep it in my back pocket. However, it's good for e-commerce where there are a subset of "important" operations, but I'd argue the idempotency key is better for a financial API like Stripe where the majority of your operations need these semantics. You also run into a problem if PUT/PATCH needs to be exactly-once instead of at least once. Not as common, but again, might be something you run into with a financial API.

stickfigure 5mo ago
There's nothing special about any particular HTTP method; it is the inherent nature of distributed systems that you can not have exactly-once semantics with a single HTTP request. To create exactly-once semantics, you need a client which retries until success and a server which prevents duplicates. The only difference between the approach I'm describing and Stripe's approach is the detail of how the client knows that it's done. "My" mechanism (notify the client that a request is a duplicate) was dominant in financial transaction processing systems before Stripe; I used probably a half dozen of them back in the day. Stripe came along and from the beginning decided "we want to compete based on making a friendly API". They succeeded, and if you ever look at Paypal's original API, it's easy to see why. It's true that repeating successful API responses makes life slightly easier for clients; they never need to check for a 409 (or whatever) response. It comes at a cost of making life quite a bit more complex on servers. Personally I don't think the tradeoff is worth it, but YMMV. If your single competitive advantage is "easy API" maybe it makes sense. If you're normal B2B, almost certainly not.
- saltcured 5mo ago
  FWIW, I enjoyed your engagement throughout the thread above, fighting the good fight. I think there is also a risk of an "easy API" when it leads to magical thinking and sloppy development. If the naive client programmer starts to think the reliability is handled for them, they may also flub the handling of the idempotence key that remains the crux here. E.g. not persisting them well enough and returning to a situation where the user can accidentally make duplicate payments just like the naive system with no idempotence feature...

mamcx 5mo ago

And a good way to convey this contract is force the clients to hash-chain the calls, so is now more clear that it should do (and even can detect the send will fail at client side)

deleted 5mo ago

[deleted]

deleted 5mo ago

[deleted]

jeremyjh 5mo ago

I really like that article, thank you for sharing it again. I wish I had read it a decade ago, or even the first time you submitted it - I had to learn some of these things the hard way. I agree with you - I don't think Stripe has made the right choices here and its unfortunate that it has inspired so many other people to make the same poor choices. I don't agree that their system is as sound as always returning 409s. I think having a short window where you return response bodies is fine, but after that they should still be sending 409s. If no one will ever actually resend a request after 24 hours how is it not fine to send 409s when they do? They've chosen to implement the more expensive choice and then not back it with the cheapest one.

deleted 5mo ago
[deleted]

reactordev 5mo ago

This is because there are countless tutorials and slop on the internet that says, no, instructs, you to handle idempotency on the client instead of where it belongs. Server-side.

malux85 5mo ago

Exactly - junior engineers haven't yet developed a good taste of "how much the system should try and deal with errors" - so they always go way overboard to guess and try and fix errors way too much. I agree with you - this is a client error - 409 and make them fix the bug on their side. Clean, reliable, simple, separation of responsibility.

kgwxd 5mo ago

> If you see a duplicate idempotency key I'm no expert but an "idempotency key" already has some major smell to it.

bcrosby95 5mo ago
Why? I've used plenty of systems that have an idempotency key. E.g. many payment processors will take an order id and won't let you charge an order id more than once. That's just an idempotency key by another name.
yencabulator 5mo ago
Then learn from the experts, maybe? https://docs.stripe.com/api/idempotent_requests https://docs.stripe.com/api/idempotent_requests

fuzzythinker 5mo ago

Thank you, like and agree with the rest of the rules