3 ms·

"Just decorate your idempotent operations" – implies to me that the idempotency of your functions is your responsibility.

CipherThrowaway 3y ago
That's one way of reading it - although I find it's at odds with the heading "Idempotency in one line of code." If it's the case that the idempotency wrapper is only for operations that are already idempotent, it makes me wonder that the docs couldn't do a better job of communicating the purpose and function. Decorate your idempotent operations.. to make them more idempotent?
lunarcave 3y ago
Not quite. The decorator [1] enforces you to provide an idempotency key, if idempotence is needed. The control-plane intercepts that and implements idempotence for you. [1] https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/
- hmeh 3y ago
  Yes, quite. Unless my function has no side-effects, idempotency will always be my responsibility. This feature is a lie and an outright hazard. It should not be called idempotency and it should have a big bold warning stating that the function may be called more than once.
  - lunarcave 3y ago
    I'd appreciate it if you could point out how the function can be called more than once. Here's the source code for the orchestrator [1] and a demonstration of it [2]. If there's an error in the source code or the copy, I'll happily retract it. [1] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/control-plane/src/modules/jobs/create-job.ts#L45 https://github.com/differentialhq/differential/blob/236ffc53... [2] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/ts-core/src/tests/idempotency/idempotency.test.ts#L4 https://github.com/differentialhq/differential/blob/236ffc53...
    - hmeh 3y ago
      Without looking at the code, I can tell you that this will reproduce one of two things: 1. Invoke function that has two side effects 2. Function does first side effect 3. Pull the computer's plug out of the wall A framework can only do one of two things: A. Invoke the function again B. Not invoke the function again If you choose A, you have at least once processing If you choose B, you have at most once processing Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one. I believe you mention exactly once processing in another comment. It's this. The massive lie that many of these types of frameworks tell is that they can manage idempotence for you. Aside from functions that are side effect free (that is, pure, that is referentially transparent, that is can be made mathematically idempotent) you cannot possibly make them idempotent. It's the two generals problem. So, I believe what you have implemented is idempotence at the sender with the reservation pattern. This is not the same as idempotence at the processor which is literally the only thing that matters. Idempotence at the sender is simply a performance (or storage, in the case of event sourcing) optimization. Please correct me if I'm wrong on any of this, but if I am, then clearly my understanding of the two generals problem and distributed systems are wrong and I'm going to have to do some significant rethinking of our system's architecture.
      - lunarcave 3y ago
        I don't think you're wrong about any of this. I just feel like we're talking over each other. You clearly demonstrate an understanding of the domain, so I'm happy to engage. After all, if we fail, we want to fail fast. Maybe we can focus the conversation on the scenario you mentioned above. > Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one. So, what I'm telling is, yes - the first side-effect (or any side-effect) that you put through the system can be made idempotent through the same tooling that makes the caller idempotent. func foo(n1) { return networkCall(n1) } func bar(n2) { return networkCall(n2) } func foobar(n1) { n2 = foo(n1) n3 = bar(n2) } func main() { foobar(42) } Given the above, the framework allows you to wrap foo, and bar all independently in higher order functions which will not issue a duplicative function calls for the same idempotency key. Therefore, you can call foobar repeatedly, and get the same result.

CipherThrowaway 3y ago

That's one way of reading it - although I find it's at odds with the heading "Idempotency in one line of code." If it's the case that the idempotency wrapper is only for operations that are already idempotent, it makes me wonder that the docs couldn't do a better job of communicating the purpose and function. Decorate your idempotent operations.. to make them more idempotent?

lunarcave 3y ago

Not quite. The decorator [1] enforces you to provide an idempotency key, if idempotence is needed. The control-plane intercepts that and implements idempotence for you. [1] https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/

hmeh 3y ago
Yes, quite. Unless my function has no side-effects, idempotency will always be my responsibility. This feature is a lie and an outright hazard. It should not be called idempotency and it should have a big bold warning stating that the function may be called more than once.
- lunarcave 3y ago
  I'd appreciate it if you could point out how the function can be called more than once. Here's the source code for the orchestrator [1] and a demonstration of it [2]. If there's an error in the source code or the copy, I'll happily retract it. [1] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/control-plane/src/modules/jobs/create-job.ts#L45 https://github.com/differentialhq/differential/blob/236ffc53... [2] https://github.com/differentialhq/differential/blob/236ffc5367fc0c8d8645fbdf41603592f7a2630d/ts-core/src/tests/idempotency/idempotency.test.ts#L4 https://github.com/differentialhq/differential/blob/236ffc53...
  - hmeh 3y ago
    Without looking at the code, I can tell you that this will reproduce one of two things: 1. Invoke function that has two side effects 2. Function does first side effect 3. Pull the computer's plug out of the wall A framework can only do one of two things: A. Invoke the function again B. Not invoke the function again If you choose A, you have at least once processing If you choose B, you have at most once processing Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one. I believe you mention exactly once processing in another comment. It's this. The massive lie that many of these types of frameworks tell is that they can manage idempotence for you. Aside from functions that are side effect free (that is, pure, that is referentially transparent, that is can be made mathematically idempotent) you cannot possibly make them idempotent. It's the two generals problem. So, I believe what you have implemented is idempotence at the sender with the reservation pattern. This is not the same as idempotence at the processor which is literally the only thing that matters. Idempotence at the sender is simply a performance (or storage, in the case of event sourcing) optimization. Please correct me if I'm wrong on any of this, but if I am, then clearly my understanding of the two generals problem and distributed systems are wrong and I'm going to have to do some significant rethinking of our system's architecture.
    - lunarcave 3y ago
      I don't think you're wrong about any of this. I just feel like we're talking over each other. You clearly demonstrate an understanding of the domain, so I'm happy to engage. After all, if we fail, we want to fail fast. Maybe we can focus the conversation on the scenario you mentioned above. > Neither of these are exactly once, therefore, neither one of these are idempotent unless the first side effect that the function performs is also idempotent. That is, you must be able to retry that one. So, what I'm telling is, yes - the first side-effect (or any side-effect) that you put through the system can be made idempotent through the same tooling that makes the caller idempotent. func foo(n1) { return networkCall(n1) } func bar(n2) { return networkCall(n2) } func foobar(n1) { n2 = foo(n1) n3 = bar(n2) } func main() { foobar(42) } Given the above, the framework allows you to wrap foo, and bar all independently in higher order functions which will not issue a duplicative function calls for the same idempotency key. Therefore, you can call foobar repeatedly, and get the same result.