5 ms·

Googler, opinions are my own. I work in payment and idempotency is always a discussion point. This draft links to one of our docs (though the draft points to

by kyrra 5y ago

Googler, opinions are my own. I work in payment and idempotency is always a discussion point. This draft links to one of our docs (though the draft points to a 404 URL, the correct link being[0]). As the draft calls out, Google Standard Payments uses the "requestId" field as our idempontency key, which is part of the request body, rather than in the HTTP header. We did this specifically to decouple the protocol from the behavior of the request. We could take the existing definitions and put that payload into any other protocol and it would work without having to use that protocol's sidechannel data (aka, the HTTP header). Our design makes idempontency part of the application layer, rather than any middlebox or load balancer. [0] https://developers.google.com/standard-payments/guides/connectivity/protocol-standards#request_idempotency https://developers.google.com/standard-payments/guides/conne...

herodoturtle 5y ago
Thanks for the insightful comment; makes good sense. Could you kindly share your view on the security implications of this approach - in particular with reference to the security context outlined in section 5 of the IETF draft: > For idempotent request handling, the resources MAY make use of the value in the idempotency key to look up a cache or a persistent store for duplicate requests matching the key. If the resource does not validate the value of the idempotency key prior to performing such a lookup, it MAY lead to various forms of security attacks and compromise.
- nivertech 5y ago
  Yes, that's the case if you use it for unauthenticated users or public API calls. For that reason I do not support idempotent retries for public API calls. For the authenitcated users, you cache/store the idempotency keys scoped by the currently authenticated user, i.e: {user_id, idempotency_key} You can add more parts to thie key for better security, i.e.: {user_id, api_call_name, idempotency_key} or even {user_id, api_call_name, input_args_hash, idempotency_key} etc.
hit8run 5y ago
Interesting and I totally get your point. What protocols do you support besides HTTP?
- dorianmariefr 5y ago
  Probably all kinds of RPC
- kyrra 5y ago
  There is the possibility of grpc, but there hasn't been a case for it yet (hence why none of our docs even mention it). Google also loves talking grpc/stubby internally, so any Google internal application to application calls would use that.
  - vettyvignesh 5y ago
    Pardon my ignorance but why does it matter that you're using RPC/graphQL or REST based service, isn't the underlying protocol still HTTP? So this spec would still be applicable, right?
    - denysvitali 5y ago
      AFAIK, in Google's case the end application doesn't really see the "GRPC HTTP headers" but instead they convert an incoming HTTP request to one of their frontends and route it to multiple backends (sending effectively the HTTP body serialized via GRPC), and then the frontends will simply re-create the response by unifying them. Not a Googler, this is what I understood from reading similar things over multiple posts. Feel free to correct me.
    - mcqueenjordan 5y ago
      The underlying protocol is not necessarily HTTP.
      - staticassertion 5y ago
        I think it is necessarily HTTP/2, but would be happy to learn I'm mistaken - it's not something I'm overly confident about. https://developers.googleblog.com/2015/02/introducing-grpc-new-open-source-http2.html https://developers.googleblog.com/2015/02/introducing-grpc-n...
brandur 5y ago
A nice feature of keeping the idempotency key separate from the payload is that a service like Stripe can build tools to help users with idempotency even if the user has no idea what an idempotency key is. For example, take a look at stripe-go's implementation, which automatically tags a request with a key if the user didn't specify one: https://github.com/stripe/stripe-go/blob/67034d2205c0240ade90e920d4b17d39e07a9c04/stripe.go#L344 https://github.com/stripe/stripe-go/blob/67034d2205c0240ade9... This works for all mutating requests, and is useful because the built-in retry system will automatically reuse the same key that was generated. Users can get the benefits of idempotency without really having to understand very well what's going on under the hood. I suppose you could still do that by munging each request body, but IMO it's a nice feature to make sure that requests are the same as what the user specified. Also note that in practice the implementations are probably not that wildly different under the hood — despite being in a header, Stripe's idempotency is still being handled by the same application stack which processes the payment (i.e. not a middle box or load balancer).
- bbss 5y ago
  The downside with “automagically” trying to handle idempotency is users may not be aware of it and retries may happen across different processes (maybe they are running their application on k8s with multiple pods), which doesn’t work with stripes default behaviour. IMO the idempotency key should be required to be set and make the user aware that they need to handle retries properly.
  - brandur 5y ago
    You might find differing philosophies depending on where you look, but a recurring theme you'll find with Stripe's is that they try to make it as easy as possible to get an integration up and running. When you're building out a payment integration, you're already awash in non-trivial concepts that are probably new and novel to you, so things that can be abstracted away for the time being to make things easier generally are. In the situation you describe, I think it would make more sense to just retry the call a couple times from the same pod so you don't have the sizable overhead of discarding it and creating a new one for every failure, in which case the automatic keys would work fine. And if there's a really good reason they're not, setting the keys manually is very easy. At some point if you're far enough off the beaten path, you have to expect to read some docs.
felixhuttmann 5y ago
Strongly agree that idempotency should be understood as part of the application, and not as part of the transport. I find it insightful to think about a transport layer that is much less reliable than the internet: traditional/snail mail. In a proper business communication, every invoice has an invoice number such that the recipient can know whether they processed that invoice previously or not.
- eweise 5y ago
  An invoice number is not the same as an idempotency-key. The idempotency-key is create by the recipient, sent to server and an invoice number is returned. Invoice number is clearly part of the application layer since its domain specific but idempotency-key is a generalized concept that prevents duplicate actions from occurring when there is no other key.
  - mikem170 5y ago
    Like a PO (purchase order) number?
  - nivertech 5y ago
    These are 2 different kinds of idempotency: 1. request (also command/mutation) idempotency (using Idempotency-Key in HTTP, or clientMutationId in GraphQL/Relay) 2. entity idempotency (using invoice_id) (1) is more generic one and can be applied to any mutation with a side-effect, for example sending an email. It can also work with server-side generated IDs. It can be implemented as a generic middleware via cache or DB with TTL. That's what Stripe is doing. (2) is more application-specific, for mutations actually changing a state (usually a DB row/document, or an entity managed by another service). Here the entity id should have a uniqueness constraint in the DB, and you should use either a semantic key (i.e. customer name), or a client-side generated ID like for example UUID. I use a combination of both, to reduce a load on the DB while still having the same guarantees as using DB only.
oneplane 5y ago
Making idempotency a protocol-level thing would be problematic with the many connection-breaking systems, even if the browser-ingress channel is TLS encrypted. You'd have the ingress proxy, maybe some IDS/IPS, maybe Cloudflare, maybe some router or gateway, some service proxy, who knows. If any of them decide to drop any headers, rewrite headers, or mis-parse them you'd lose the idempotency completely. It does however make a difference if not the body, but the protocol-level request is supposed to be the idempotent one, but I don't know in what kind of a scenario that would apply.
tankerdude 5y ago
That’s dead on what I’ve been doing on my projects for some time. I also track it with a correlation id once it reaches my network so that I can put all logging and context into ELK for my aggregated logging. I can then see everything that that request did in my entire system.
donavanm 5y ago
Very similar to the approach AWS takes with ‘clientToken’ being an API parameter, not transport/protocol header, for operationz that need it. If clients dont specify a value the SDK will populate the idempotency token and handle retries etc transparently. The approach is that API/SDK/client semantics should be similar across services, but very few users care about actual transport or “on the wire” serialization. Disclaimer: principal at AWS, but just general observations on this topic.
no_circuit 5y ago
I've worked on fintech as well. All of my full time jobs have dealt with both idempotency and API error code issues across both trading fintech as well as consumer social apps. App-level error codes, which includes idempotency, seem to be best handled in app request bodies, not the enveloping protocol layer. In trading fintech there is the FIX API [0] which has been around for a while. It has a "Client Order Id" field which gets mapped to an (Exchange) "Order Id" field. I've also worked on the Twitter apps. The API spec seems fairly clean [1], but at least when I was there some time ago, there were edge cases that both the HTTP error and app error codes had to be combined to find the true error in order to take the correct action and show an appropriate error message to the user. Idempotency is a very app-specific error condition. So I'm definitely in the group of developers that would not want to see nor code for app errors at a transport like HTTP level. I feel that the header idea and spec has good intentions, but does not take into consideration the developer experience as an API grows more complex. It also forces changes at the app messaging level if one wants to support different transport types (UDP unicast or multicast, gRPC [2], etc.). [0] https://en.wikipedia.org/wiki/Financial_Information_eXchange https://en.wikipedia.org/wiki/Financial_Information_eXchange [1] https://developer.twitter.com/en/support/twitter-api/error-troubleshooting https://developer.twitter.com/en/support/twitter-api/error-t... [2] https://grpc.github.io/grpc/core/md_doc_statuscodes.html https://grpc.github.io/grpc/core/md_doc_statuscodes.html

by kyrra 5y ago

herodoturtle 5y ago

Thanks for the insightful comment; makes good sense. Could you kindly share your view on the security implications of this approach - in particular with reference to the security context outlined in section 5 of the IETF draft: > For idempotent request handling, the resources MAY make use of the value in the idempotency key to look up a cache or a persistent store for duplicate requests matching the key. If the resource does not validate the value of the idempotency key prior to performing such a lookup, it MAY lead to various forms of security attacks and compromise.

nivertech 5y ago
Yes, that's the case if you use it for unauthenticated users or public API calls. For that reason I do not support idempotent retries for public API calls. For the authenitcated users, you cache/store the idempotency keys scoped by the currently authenticated user, i.e: {user_id, idempotency_key} You can add more parts to thie key for better security, i.e.: {user_id, api_call_name, idempotency_key} or even {user_id, api_call_name, input_args_hash, idempotency_key} etc.

hit8run 5y ago

Interesting and I totally get your point. What protocols do you support besides HTTP?

dorianmariefr 5y ago
Probably all kinds of RPC
kyrra 5y ago
There is the possibility of grpc, but there hasn't been a case for it yet (hence why none of our docs even mention it). Google also loves talking grpc/stubby internally, so any Google internal application to application calls would use that.
- vettyvignesh 5y ago
  Pardon my ignorance but why does it matter that you're using RPC/graphQL or REST based service, isn't the underlying protocol still HTTP? So this spec would still be applicable, right?
  - denysvitali 5y ago
    AFAIK, in Google's case the end application doesn't really see the "GRPC HTTP headers" but instead they convert an incoming HTTP request to one of their frontends and route it to multiple backends (sending effectively the HTTP body serialized via GRPC), and then the frontends will simply re-create the response by unifying them. Not a Googler, this is what I understood from reading similar things over multiple posts. Feel free to correct me.
  - mcqueenjordan 5y ago
    The underlying protocol is not necessarily HTTP.
    - staticassertion 5y ago
      I think it is necessarily HTTP/2, but would be happy to learn I'm mistaken - it's not something I'm overly confident about. https://developers.googleblog.com/2015/02/introducing-grpc-new-open-source-http2.html https://developers.googleblog.com/2015/02/introducing-grpc-n...

brandur 5y ago

A nice feature of keeping the idempotency key separate from the payload is that a service like Stripe can build tools to help users with idempotency even if the user has no idea what an idempotency key is. For example, take a look at stripe-go's implementation, which automatically tags a request with a key if the user didn't specify one: https://github.com/stripe/stripe-go/blob/67034d2205c0240ade90e920d4b17d39e07a9c04/stripe.go#L344 https://github.com/stripe/stripe-go/blob/67034d2205c0240ade9... This works for all mutating requests, and is useful because the built-in retry system will automatically reuse the same key that was generated. Users can get the benefits of idempotency without really having to understand very well what's going on under the hood. I suppose you could still do that by munging each request body, but IMO it's a nice feature to make sure that requests are the same as what the user specified. Also note that in practice the implementations are probably not that wildly different under the hood — despite being in a header, Stripe's idempotency is still being handled by the same application stack which processes the payment (i.e. not a middle box or load balancer).

bbss 5y ago
The downside with “automagically” trying to handle idempotency is users may not be aware of it and retries may happen across different processes (maybe they are running their application on k8s with multiple pods), which doesn’t work with stripes default behaviour. IMO the idempotency key should be required to be set and make the user aware that they need to handle retries properly.
- brandur 5y ago
  You might find differing philosophies depending on where you look, but a recurring theme you'll find with Stripe's is that they try to make it as easy as possible to get an integration up and running. When you're building out a payment integration, you're already awash in non-trivial concepts that are probably new and novel to you, so things that can be abstracted away for the time being to make things easier generally are. In the situation you describe, I think it would make more sense to just retry the call a couple times from the same pod so you don't have the sizable overhead of discarding it and creating a new one for every failure, in which case the automatic keys would work fine. And if there's a really good reason they're not, setting the keys manually is very easy. At some point if you're far enough off the beaten path, you have to expect to read some docs.

felixhuttmann 5y ago

Strongly agree that idempotency should be understood as part of the application, and not as part of the transport. I find it insightful to think about a transport layer that is much less reliable than the internet: traditional/snail mail. In a proper business communication, every invoice has an invoice number such that the recipient can know whether they processed that invoice previously or not.

eweise 5y ago
An invoice number is not the same as an idempotency-key. The idempotency-key is create by the recipient, sent to server and an invoice number is returned. Invoice number is clearly part of the application layer since its domain specific but idempotency-key is a generalized concept that prevents duplicate actions from occurring when there is no other key.
- mikem170 5y ago
  Like a PO (purchase order) number?
- nivertech 5y ago
  These are 2 different kinds of idempotency: 1. request (also command/mutation) idempotency (using Idempotency-Key in HTTP, or clientMutationId in GraphQL/Relay) 2. entity idempotency (using invoice_id) (1) is more generic one and can be applied to any mutation with a side-effect, for example sending an email. It can also work with server-side generated IDs. It can be implemented as a generic middleware via cache or DB with TTL. That's what Stripe is doing. (2) is more application-specific, for mutations actually changing a state (usually a DB row/document, or an entity managed by another service). Here the entity id should have a uniqueness constraint in the DB, and you should use either a semantic key (i.e. customer name), or a client-side generated ID like for example UUID. I use a combination of both, to reduce a load on the DB while still having the same guarantees as using DB only.

oneplane 5y ago

Making idempotency a protocol-level thing would be problematic with the many connection-breaking systems, even if the browser-ingress channel is TLS encrypted. You'd have the ingress proxy, maybe some IDS/IPS, maybe Cloudflare, maybe some router or gateway, some service proxy, who knows. If any of them decide to drop any headers, rewrite headers, or mis-parse them you'd lose the idempotency completely. It does however make a difference if not the body, but the protocol-level request is supposed to be the idempotent one, but I don't know in what kind of a scenario that would apply.

tankerdude 5y ago

That’s dead on what I’ve been doing on my projects for some time. I also track it with a correlation id once it reaches my network so that I can put all logging and context into ELK for my aggregated logging. I can then see everything that that request did in my entire system.

donavanm 5y ago

Very similar to the approach AWS takes with ‘clientToken’ being an API parameter, not transport/protocol header, for operationz that need it. If clients dont specify a value the SDK will populate the idempotency token and handle retries etc transparently. The approach is that API/SDK/client semantics should be similar across services, but very few users care about actual transport or “on the wire” serialization. Disclaimer: principal at AWS, but just general observations on this topic.

no_circuit 5y ago

I've worked on fintech as well. All of my full time jobs have dealt with both idempotency and API error code issues across both trading fintech as well as consumer social apps. App-level error codes, which includes idempotency, seem to be best handled in app request bodies, not the enveloping protocol layer. In trading fintech there is the FIX API [0] which has been around for a while. It has a "Client Order Id" field which gets mapped to an (Exchange) "Order Id" field. I've also worked on the Twitter apps. The API spec seems fairly clean [1], but at least when I was there some time ago, there were edge cases that both the HTTP error and app error codes had to be combined to find the true error in order to take the correct action and show an appropriate error message to the user. Idempotency is a very app-specific error condition. So I'm definitely in the group of developers that would not want to see nor code for app errors at a transport like HTTP level. I feel that the header idea and spec has good intentions, but does not take into consideration the developer experience as an API grows more complex. It also forces changes at the app messaging level if one wants to support different transport types (UDP unicast or multicast, gRPC [2], etc.). [0] https://en.wikipedia.org/wiki/Financial_Information_eXchange https://en.wikipedia.org/wiki/Financial_Information_eXchange [1] https://developer.twitter.com/en/support/twitter-api/error-troubleshooting https://developer.twitter.com/en/support/twitter-api/error-t... [2] https://grpc.github.io/grpc/core/md_doc_statuscodes.html https://grpc.github.io/grpc/core/md_doc_statuscodes.html