4 ms·

You could have some side channel with an identifier for the RPC. If the server receives duplicates of one RPC, it ignores the extras. It would require some cli

You could have some side channel with an identifier for the RPC. If the server receives duplicates of one RPC, it ignores the extras. It would require some client support, though. Alternatively, if the RPCs have the same arguments, the framework could choose to ignore seemingly identical ones, though that has its own issues.

CipherThrowaway 3y ago
Neither method works. If the framework records the occurrence of the call before the effect of the function, you achieve at-most-once semantics. If it records the call after the effect, you get at-least-once. The framework might perform its idempotency bookkeeping within the same transaction boundary as the function's side-effects, but this means the function implementation is no longer a black-box. E.g. you can no longer perform arbitrary side-effects while preserving the idempotency guarantees of the framework.
- jitl 3y ago
  It’s at-least-once. From https://docs.differential.dev/advanced/compute-recovery/ https://docs.differential.dev/advanced/compute-recovery/ > If a machine fails to send any heartbeats within an interval (default 90 seconds): > It is marked as unhealthy, and Differential will not send any new requests to it. > The functions in progress are marked as failed, and Differential will retry them on a healthy worker. I would guess that “idempotent” functions in the system also take a lease out on the idempotency key. Perhaps they release the lease on failure since they can observe errors thrown, and they commit the key as consumed after success. ¯\_(ツ)_/¯ the docs are not clear on these semantics!
  - Salgat 3y ago
    That's how we handle idempotency. It's basically a mutex on the idempotency key with a timeout (in our case using redis with redlock since it's a distributed system). Once the command finishes, the key is marked as handled and the lock is released. At that point any future or queued requests with the same key immediately return.
    - CipherThrowaway 3y ago
      It sounds to me like there are scenarios where calls to, and certainly effects of, "idempotent" functions can take place twice. I haven't looked at Redlock for a while, but I'm assuming you're all across the historical objections to it from distributed systems researchers: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
      - fweimer 3y ago
        Is there really consensus that practical systems need to have lock lease timeouts?
        jitl 3y ago
        It’s a choice between at-least-once and at-most-once right? Without a timeout you get at-most-once since the lease holder may die off and never complete the task, and so the task of a completed 0 times. I’m building a system right now and we’re going to use lease/heartbeats etc etc to elect leaders and whatnot, all this discussion seems very practical to me.
        Salgat 3y ago
        Exactly. It comes down to whether the consequences of these race conditions are acceptable or not. Sometimes (known and understood) race conditions are okay for performance reasons. Shoot, that's the whole philosophy behind eventual consistency. Sometimes they aren't, such as bank transactions. It's all situational.
        CipherThrowaway 3y ago
        Eventual consistency isn't a philosophy, and it's not the idea that race conditions are acceptable. It's a consistency model with specific properties and guarantees. These races we're talking about don't produce eventual consistency, but inconsistency. Different systems require different levels of correctness, but if you're not familiar with the underlying theory then your trade-offs are going to be uninformed decisions rather than informed ones.
  - lunarcave 3y ago
    Ok, I think we could do a better job of explaining the semantics. The system does not implement idempotency by default. If something that's not marked as "idempotent" fails, it does try again in a different machine. If the function is marked as idempotent [1] then the system ensure at-most-once semantics by only issuing the call once and only once to a worker who can process the call. [1] https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/
reissbaker 3y ago
Yeah, this is what it looks like in the docs; you choose an "idempotency key": https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/

CipherThrowaway 3y ago

Neither method works. If the framework records the occurrence of the call before the effect of the function, you achieve at-most-once semantics. If it records the call after the effect, you get at-least-once. The framework might perform its idempotency bookkeeping within the same transaction boundary as the function's side-effects, but this means the function implementation is no longer a black-box. E.g. you can no longer perform arbitrary side-effects while preserving the idempotency guarantees of the framework.

jitl 3y ago
It’s at-least-once. From https://docs.differential.dev/advanced/compute-recovery/ https://docs.differential.dev/advanced/compute-recovery/ > If a machine fails to send any heartbeats within an interval (default 90 seconds): > It is marked as unhealthy, and Differential will not send any new requests to it. > The functions in progress are marked as failed, and Differential will retry them on a healthy worker. I would guess that “idempotent” functions in the system also take a lease out on the idempotency key. Perhaps they release the lease on failure since they can observe errors thrown, and they commit the key as consumed after success. ¯\_(ツ)_/¯ the docs are not clear on these semantics!
- Salgat 3y ago
  That's how we handle idempotency. It's basically a mutex on the idempotency key with a timeout (in our case using redis with redlock since it's a distributed system). Once the command finishes, the key is marked as handled and the lock is released. At that point any future or queued requests with the same key immediately return.
  - CipherThrowaway 3y ago
    It sounds to me like there are scenarios where calls to, and certainly effects of, "idempotent" functions can take place twice. I haven't looked at Redlock for a while, but I'm assuming you're all across the historical objections to it from distributed systems researchers: https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html https://martin.kleppmann.com/2016/02/08/how-to-do-distribute...
    - fweimer 3y ago
      Is there really consensus that practical systems need to have lock lease timeouts?
      - jitl 3y ago
        It’s a choice between at-least-once and at-most-once right? Without a timeout you get at-most-once since the lease holder may die off and never complete the task, and so the task of a completed 0 times. I’m building a system right now and we’re going to use lease/heartbeats etc etc to elect leaders and whatnot, all this discussion seems very practical to me.
      - Salgat 3y ago
        Exactly. It comes down to whether the consequences of these race conditions are acceptable or not. Sometimes (known and understood) race conditions are okay for performance reasons. Shoot, that's the whole philosophy behind eventual consistency. Sometimes they aren't, such as bank transactions. It's all situational.
        CipherThrowaway 3y ago
        Eventual consistency isn't a philosophy, and it's not the idea that race conditions are acceptable. It's a consistency model with specific properties and guarantees. These races we're talking about don't produce eventual consistency, but inconsistency. Different systems require different levels of correctness, but if you're not familiar with the underlying theory then your trade-offs are going to be uninformed decisions rather than informed ones.
- lunarcave 3y ago
  Ok, I think we could do a better job of explaining the semantics. The system does not implement idempotency by default. If something that's not marked as "idempotent" fails, it does try again in a different machine. If the function is marked as idempotent [1] then the system ensure at-most-once semantics by only issuing the call once and only once to a worker who can process the call. [1] https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/

reissbaker 3y ago

Yeah, this is what it looks like in the docs; you choose an "idempotency key": https://docs.differential.dev/advanced/idempotency/ https://docs.differential.dev/advanced/idempotency/