4 ms·

How does the idempotency work? I don't understand how functions can be made idempotent without changing or instrumenting their implementation for end-to-end ide

How does the idempotency work? I don't understand how functions can be made idempotent without changing or instrumenting their implementation for end-to-end idempotency.

scarmig 3y ago
You could have some side channel with an identifier for the RPC. If the server receives duplicates of one RPC, it ignores the extras. It would require some client support, though. Alternatively, if the RPCs have the same arguments, the framework could choose to ignore seemingly identical ones, though that has its own issues.
- CipherThrowaway 3y ago
  Neither method works. If the framework records the occurrence of the call before the effect of the function, you achieve at-most-once semantics. If it records the call after the effect, you get at-least-once. The framework might perform its idempotency bookkeeping within the same transaction boundary as the function's side-effects, but this means the function implementation is no longer a black-box. E.g. you can no longer perform arbitrary side-effects while preserving the idempotency guarantees of the framework.
  - jitl 3y ago
    It’s at-least-once. From https://docs.differential.dev/advanced/compute-recovery/ https://docs.differential.dev/advanced/compute-recovery/ > If a machine fails to send any heartbeats within an interval (default 90 seconds): > It is marked as unhealthy, and Differential will not send any new requests to it. > The functions in progress are marked as failed, and Differential will retry them on a healthy worker. I would guess that “idempotent” functions in the system also take a lease out on the idempotency key. Perhaps they release the lease on failure since they can observe errors thrown, and they commit the key as consumed after success. ¯\_(ツ)_/¯ the docs are not clear on these semantics!
    - Salgat 3y ago
      That's how we handle idempotency. It's basically a mutex on the idempotency key with a timeout (in our case using redis with redlock since it's a distributed system). Once the command finishes, the key is marked as handled and the lock is released. At that point any future or queued requests with the same key immediately return.
      - CipherThrowaway 3y ago
        It sounds to me like there are scenarios where calls to, and certainly effects of, "idempotent" functions can take place twice. I haven't looked at Redlock for a while, but I'm assuming you're all across the historical objections to it from distributed systems researchers: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
        fweimer 3y ago
        Is there really consensus that practical systems need to have lock lease timeouts?
        jitl 3y ago
        It’s a choice between at-least-once and at-most-once right? Without a timeout you get at-most-once since the lease holder may die off and never complete the task, and so the task of a completed 0 times. I’m building a system right now and we’re going to use lease/heartbeats etc etc to elect leaders and whatnot, all this discussion seems very practical to me.
        Salgat 3y ago
        Exactly. It comes down to whether the consequences of these race conditions are acceptable or not. Sometimes (known and understood) race conditions are okay for performance reasons. Shoot, that's the whole philosophy behind eventual consistency. Sometimes they aren't, such as bank transactions. It's all situational.
- reissbaker 3y ago
  Yeah, this is what it looks like in the docs; you choose an "idempotency key": https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/
asplake 3y ago
"Just decorate your idempotent operations" – implies to me that the idempotency of your functions is your responsibility.
- CipherThrowaway 3y ago
  That's one way of reading it - although I find it's at odds with the heading "Idempotency in one line of code." If it's the case that the idempotency wrapper is only for operations that are already idempotent, it makes me wonder that the docs couldn't do a better job of communicating the purpose and function. Decorate your idempotent operations.. to make them more idempotent?
- lunarcave 3y ago
  Not quite. The decorator [1] enforces you to provide an idempotency key, if idempotence is needed. The control-plane intercepts that and implements idempotence for you. [1] https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/
  - hmeh 3y ago
    Yes, quite. Unless my function has no side-effects, idempotency will always be my responsibility. This feature is a lie and an outright hazard. It should not be called idempotency and it should have a big bold warning stating that the function may be called more than once.
    - lunarcave 3y ago
      I'd appreciate it if you could point out how the function can be called more than once. Here's the source code for the orchestrator [1] and a demonstration of it [2]. If there's an error in the source code or the copy, I'll happily retract it. [1] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/control-plane/src/modules/jobs/create-job.ts#L45 https://github.com/differentialhq/differential/blob/236ffc53... [2] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/ts-core/src/tests/idempotency/idempotency.test.ts#L4 https://github.com/differentialhq/differential/blob/236ffc53...
lunarcave 3y ago
Here are the docs on idempotency: https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/
- CipherThrowaway 3y ago
  You mentioned elsewhere that you're still finding the right wording for the product and its benefits. Fair enough. Personally, I think you should probably stop referring to this as idempotency. Maybe describe it as a retry policy, or as a choice between at-least-once and at-most-once. I expect most people with knowledge of - and respect for - distributed systems theory will be completely put-off by this wording. In distributed systems, the reason idempotency is such a Big Deal™ is that it can be combined with at-least-once delivery to achieve exactly-once semantics. This isn't that. What you're describing as idempotency has the potential to mislead users, especially since you use examples like credit card processing.
  - lunarcave 3y ago
    Thanks for the thoughtful reply. You might be on to something here: > Maybe describe it as a retry policy, or as a choice between at-least-once and at-most-once and we will consider changing it because I agree that conflating the concepts (or the appearance thereof) is not worth diluting the other parts of the offering. I still struggle to find the difference between this approach and the likes of Stripe [1] and Temporal [2] because in practice, it yields the same result. [1] https://docs.stripe.com/api/idempotent_requests https://docs.stripe.com/api/idempotent_requests [2] https://www.restack.io/docs/temporal-knowledge-temporal-io-idempotency https://www.restack.io/docs/temporal-knowledge-temporal-io-i...
    - CipherThrowaway 3y ago
      One difference is that you are specifically creating distributed systems middleware, where your audience is going to be using your product in the implementation of a system of their own, and is more likely to understand the terminology and wish to know the exact guarantees and trade-offs being provided. I find both those comparisons are apples to oranges: 1. Temporal does not claim, on that page, to be able to automatically make your calls idempotent. Instead, it is explaining how you - as the user - can and should write your activities to be idempotent. Take the payment code snippet: you can't just wrap `ExternalPaymentAPI.Process(paymentId, amount)` in an idempotency provider. You have to implement specific idempotency logic inside the activity. This is in line with the distributed systems theory of idempotency. 2. In these docs, Stripe describes an interface for achieving idempotency as an external caller. Since Stripe control the implementation of their own system, it is certainly possible that they have implemented their operations in a way that is truly idempotent. What your docs describe - the ability to make arbitrary operations idempotent by wrapping them - is simply not possible. I wonder whether you're focusing on the idempotency key as the thing that is wrong here. There is nothing wrong with idempotency keys themselves - they are a great way for APIs to provide idempotency. The problem is that this guarantee can't be provided by a wrapper layer, it requires the operation itself to be written in an idempotent way. This is an example of the age old "end-to-end principle" from networks class.

How does the idempotency work? I don't understand how functions can be made idempotent without changing or instrumenting their implementation for end-to-end idempotency.

scarmig 3y ago

You could have some side channel with an identifier for the RPC. If the server receives duplicates of one RPC, it ignores the extras. It would require some client support, though. Alternatively, if the RPCs have the same arguments, the framework could choose to ignore seemingly identical ones, though that has its own issues.

CipherThrowaway 3y ago
Neither method works. If the framework records the occurrence of the call before the effect of the function, you achieve at-most-once semantics. If it records the call after the effect, you get at-least-once. The framework might perform its idempotency bookkeeping within the same transaction boundary as the function's side-effects, but this means the function implementation is no longer a black-box. E.g. you can no longer perform arbitrary side-effects while preserving the idempotency guarantees of the framework.
- jitl 3y ago
  It’s at-least-once. From https://docs.differential.dev/advanced/compute-recovery/ https://docs.differential.dev/advanced/compute-recovery/ > If a machine fails to send any heartbeats within an interval (default 90 seconds): > It is marked as unhealthy, and Differential will not send any new requests to it. > The functions in progress are marked as failed, and Differential will retry them on a healthy worker. I would guess that “idempotent” functions in the system also take a lease out on the idempotency key. Perhaps they release the lease on failure since they can observe errors thrown, and they commit the key as consumed after success. ¯\_(ツ)_/¯ the docs are not clear on these semantics!
  - Salgat 3y ago
    That's how we handle idempotency. It's basically a mutex on the idempotency key with a timeout (in our case using redis with redlock since it's a distributed system). Once the command finishes, the key is marked as handled and the lock is released. At that point any future or queued requests with the same key immediately return.
    - CipherThrowaway 3y ago
      It sounds to me like there are scenarios where calls to, and certainly effects of, "idempotent" functions can take place twice. I haven't looked at Redlock for a while, but I'm assuming you're all across the historical objections to it from distributed systems researchers: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
      - fweimer 3y ago
        Is there really consensus that practical systems need to have lock lease timeouts?
        jitl 3y ago
        It’s a choice between at-least-once and at-most-once right? Without a timeout you get at-most-once since the lease holder may die off and never complete the task, and so the task of a completed 0 times. I’m building a system right now and we’re going to use lease/heartbeats etc etc to elect leaders and whatnot, all this discussion seems very practical to me.
        Salgat 3y ago
        Exactly. It comes down to whether the consequences of these race conditions are acceptable or not. Sometimes (known and understood) race conditions are okay for performance reasons. Shoot, that's the whole philosophy behind eventual consistency. Sometimes they aren't, such as bank transactions. It's all situational.
reissbaker 3y ago
Yeah, this is what it looks like in the docs; you choose an "idempotency key": https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/

asplake 3y ago

"Just decorate your idempotent operations" – implies to me that the idempotency of your functions is your responsibility.

CipherThrowaway 3y ago
That's one way of reading it - although I find it's at odds with the heading "Idempotency in one line of code." If it's the case that the idempotency wrapper is only for operations that are already idempotent, it makes me wonder that the docs couldn't do a better job of communicating the purpose and function. Decorate your idempotent operations.. to make them more idempotent?
lunarcave 3y ago
Not quite. The decorator [1] enforces you to provide an idempotency key, if idempotence is needed. The control-plane intercepts that and implements idempotence for you. [1] https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/
- hmeh 3y ago
  Yes, quite. Unless my function has no side-effects, idempotency will always be my responsibility. This feature is a lie and an outright hazard. It should not be called idempotency and it should have a big bold warning stating that the function may be called more than once.
  - lunarcave 3y ago
    I'd appreciate it if you could point out how the function can be called more than once. Here's the source code for the orchestrator [1] and a demonstration of it [2]. If there's an error in the source code or the copy, I'll happily retract it. [1] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/control-plane/src/modules/jobs/create-job.ts#L45 https://github.com/differentialhq/differential/blob/236ffc53... [2] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/ts-core/src/tests/idempotency/idempotency.test.ts#L4 https://github.com/differentialhq/differential/blob/236ffc53...

lunarcave 3y ago

Here are the docs on idempotency: https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/

CipherThrowaway 3y ago
You mentioned elsewhere that you're still finding the right wording for the product and its benefits. Fair enough. Personally, I think you should probably stop referring to this as idempotency. Maybe describe it as a retry policy, or as a choice between at-least-once and at-most-once. I expect most people with knowledge of - and respect for - distributed systems theory will be completely put-off by this wording. In distributed systems, the reason idempotency is such a Big Deal™ is that it can be combined with at-least-once delivery to achieve exactly-once semantics. This isn't that. What you're describing as idempotency has the potential to mislead users, especially since you use examples like credit card processing.
- lunarcave 3y ago
  Thanks for the thoughtful reply. You might be on to something here: > Maybe describe it as a retry policy, or as a choice between at-least-once and at-most-once and we will consider changing it because I agree that conflating the concepts (or the appearance thereof) is not worth diluting the other parts of the offering. I still struggle to find the difference between this approach and the likes of Stripe [1] and Temporal [2] because in practice, it yields the same result. [1] https://docs.stripe.com/api/idempotent_requests https://docs.stripe.com/api/idempotent_requests [2] https://www.restack.io/docs/temporal-knowledge-temporal-io-idempotency https://www.restack.io/docs/temporal-knowledge-temporal-io-i...
  - CipherThrowaway 3y ago
    One difference is that you are specifically creating distributed systems middleware, where your audience is going to be using your product in the implementation of a system of their own, and is more likely to understand the terminology and wish to know the exact guarantees and trade-offs being provided. I find both those comparisons are apples to oranges: 1. Temporal does not claim, on that page, to be able to automatically make your calls idempotent. Instead, it is explaining how you - as the user - can and should write your activities to be idempotent. Take the payment code snippet: you can't just wrap `ExternalPaymentAPI.Process(paymentId, amount)` in an idempotency provider. You have to implement specific idempotency logic inside the activity. This is in line with the distributed systems theory of idempotency. 2. In these docs, Stripe describes an interface for achieving idempotency as an external caller. Since Stripe control the implementation of their own system, it is certainly possible that they have implemented their operations in a way that is truly idempotent. What your docs describe - the ability to make arbitrary operations idempotent by wrapping them - is simply not possible. I wonder whether you're focusing on the idempotency key as the thing that is wrong here. There is nothing wrong with idempotency keys themselves - they are a great way for APIs to provide idempotency. The problem is that this guarantee can't be provided by a wrapper layer, it requires the operation itself to be written in an idempotent way. This is an example of the age old "end-to-end principle" from networks class.