4 ms·

It seems to me that you could build a protocol where normal syncs happen through webhooks where each webhook event refers to the event id of an immediately prev

It seems to me that you could build a protocol where normal syncs happen through webhooks where each webhook event refers to the event id of an immediately previous event. If the system receiving notifications doesn’t have that event, it makes an API request for all events between the latest event it has and the one it just received.

williamdclt 5y ago
That is pretty much what TCP does, except that it doesn't "request" missing packets: the client just acknowledges the ID of the last packet it was happy with. It requires a sequence though, so that you know that the client can know that packet it just received isn't the one it expected, but you could build that with a chain of IDs like you're proposing.
aboodman 5y ago
It the system is moving fast this is still somewhat complicated to implement robustly because by the time the "catchup" request to t0 returns, more time has passed and more events have happened, so you still can't resume consuming the webhooks. To be correct with such a system you have to be prepared to queue the incoming webhook events, do the catchup query, then replay the queued events.
- MrStonedOne 5y ago
  only if the order of events matters.
  - fiddlerwoaroof 5y ago
    Yeah, if the events can be handled idempotently and this handling properly accounts for time, you don’t have to stop processing webhooks while filling in the gaps.
    - aboodman 5y ago
      OK but, it usually matters. If the events represent changes to some object, those changes almost always have to be handled in order if you are to arrive at the same end state as the source. edit: Also, idempotent is not the right term here. Idempotent just would mean the event could be handled by the receiver multiple times w/o changing the meaning. If you need the events to be applicable out of order, then you need them to be commutative. This is a much more difficult property to ensure and in practice, I am guessing, almost non-existent in deployed webhook APIs.
      - fiddlerwoaroof 5y ago
        I think that’s right re. idempotency. However, I think if each webhook is a statement of the state of an object at time t (using some monotonic definition of time, like a Lamport clock), commutativity is trivial: compare the time in the webhook event to the time in your local database, and only update your database if the webhook event is newer.

williamdclt 5y ago

That is pretty much what TCP does, except that it doesn't "request" missing packets: the client just acknowledges the ID of the last packet it was happy with. It requires a sequence though, so that you know that the client can know that packet it just received isn't the one it expected, but you could build that with a chain of IDs like you're proposing.

aboodman 5y ago

It the system is moving fast this is still somewhat complicated to implement robustly because by the time the "catchup" request to t0 returns, more time has passed and more events have happened, so you still can't resume consuming the webhooks. To be correct with such a system you have to be prepared to queue the incoming webhook events, do the catchup query, then replay the queued events.

MrStonedOne 5y ago
only if the order of events matters.
- fiddlerwoaroof 5y ago
  Yeah, if the events can be handled idempotently and this handling properly accounts for time, you don’t have to stop processing webhooks while filling in the gaps.
  - aboodman 5y ago
    OK but, it usually matters. If the events represent changes to some object, those changes almost always have to be handled in order if you are to arrive at the same end state as the source. edit: Also, idempotent is not the right term here. Idempotent just would mean the event could be handled by the receiver multiple times w/o changing the meaning. If you need the events to be applicable out of order, then you need them to be commutative. This is a much more difficult property to ensure and in practice, I am guessing, almost non-existent in deployed webhook APIs.
    - fiddlerwoaroof 5y ago
      I think that’s right re. idempotency. However, I think if each webhook is a statement of the state of an object at time t (using some monotonic definition of time, like a Lamport clock), commutativity is trivial: compare the time in the webhook event to the time in your local database, and only update your database if the webhook event is newer.