5 ms·

Webhooks are fun to think about. A couple more issues off the top of my head: * Ordering. Since network requests can take variable amounts of time, how do you

Webhooks are fun to think about. A couple more issues off the top of my head: * Ordering. Since network requests can take variable amounts of time, how do you ensure those two "foo.updated" events are processed in order, or that the receiver can tell their order? Especially something to consider if the webhooks will retry a few times on intermittent failures. * Consistency. Always a concern in distributed microservice land, but maybe more acute when generating webhooks right as things are updated: if the receiver uses the webhook to make an API request right back into the system, will the API have the same view of the data? * DDoS. How do you make sure the webhook destination URL is owned by the subscriber? If your system can generate high volumes of webhook traffic, that could be a problem. * Infinite loops. A silly one, but the user could conceivably point the webhook at a URL of the system that sends the webhook, in such a way that it will cause a new webhookable event to be generated.

sarchertech 3y ago
Ordering is something that everyone always forgets. The only way to actually guarantee they are sent and received in order is to use a lock on whatever subset of messages you care about being ordered. Then only process one of those at a time. At my last company I built an ordered transaction outbox to handle this. Each message has a partition key. When the worker picks up a new message to process it locks the partition key, so that it’s the only worker processing any messages with that partition key. Different messages use different things for their partition key. For some messages where we only care about the order within a given order (it’s an order management system) we use the order_id. For other messages where we care about global ordering we use the tenant_id etc… You could just use tenant_id for every message, but by using the most granular partition that you can, you can get a lot more parallelism.
- foobiekr 3y ago
  Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases. You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model). That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier. The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing. Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.
  - Toutouxc 3y ago
    I would really like to understand the second paragraph, but I don’t understand a single sentence. What exactly are you proposing? What’s a state compressed reporter?
    - 15457345234 3y ago
      Their comment makes no real sense to me either. I can't tell from looking at their comment history whether or not they're AI, but their profile says 'I am nobody.' It's a strange world.
    - foobiekr 3y ago
      The reporter is the system that sends updates to your external observers; observers register with the reporter and the reporter facilitates their consumption of events. In the case of a webhook, this is the first tier (the actual event store) that individual webhooks contexts are following (on the service side). You need these to be per-outbound-destination to avoid head of line blocking. I am proposing that people use state synchronization paradigms and not simple event replication for their notification systems. Since the remote side is not within your control (typically) the state compression all needs to be service side. If you want to explore systems like this, you should start with the basics of how routing protocol peering actually works inside routing protocol implementations. In that context, you have dozens to hundreds of fast and slow consumers that are trying to converge with what is, basically a forever changing data set through the exchange of adds, deletes, and updates with peers. Any given downstream peer may block for extended periods of time, they may resume, and so on, and you cannot buffer significant amounts of data per peer (since the stack _must_ consume updates from its own peers). Alternately, start looking into how databases sync with one another beyond the simple replicated transaction log sense.
  - sarchertech 3y ago
    In many cases none of that is possible because you aren’t designing the entire system. Either because it’s already been partially implemented, or because you’re modeling a system that does require ordering with no drops. And whether webhooks are a good choice or not, they are often required for one reason or another.
- dalore 3y ago
  Could you not timestamp your message and then let the client do the ordering. Granted they might get messages out of order, but I never promised them their webhooks would be in order.
  - devoutsalsa 3y ago
    Regular timestamps can only be used reliably for ordering if all the timestamps come from a single system. Different systems have different clocks, so they'll never be precisely in sync. You can also have timestamp collisions, which makes ordering ambiguous.
    - chrsig 3y ago
      This is assuming the use of a monotonic clock. the system wall clock may jump backwards in time. Itt also assumes that only one id can be generated per millisecond (or whatever clock resolution). In the case of multiple threads or processes on multiple cores, it becomes more likely that there will be a collision. Timestamps don't make for a great ordering key.
  - paulddraper 3y ago
    Yes, you can push the problem to your users.
  - skeeter2020 3y ago
    Timestamps from where? There is not necessarily going to be a central authority or processor who can guarantee these are in sync across distributed systems
- paulddraper 3y ago
  > At my last company I built an ordered transaction outbox to handle this. Almost all message queue/event streaming systems support FIFO (SQS, RabbitMQ, ActiveMQ, Kafka, Kinesis). If you use one of these, it's easy to solve ordering.
  - skeeter2020 3y ago
    This is just for arrival order though. The GP mentions transaction-based operations; handling this without massive hits to parallel processing is not an easy problem to solve.
    - paulddraper 3y ago
      ? I'm not sure I understand. You choose an FIFO key, e.g. order ID. Any message with that key is processed serially. Ordering between keys is unspecified. It is important to choose a key that guarantees sufficient ordering (for correctness), but not too much (for performance).
      - ignoramous 3y ago
        > You choose an FIFO key, e.g. order ID. This is the part of the system that is doing most of the heavy lifting. Ex A, The struggles of Elasticsearch: https://archive.is/rm1UI https://archive.is/rm1UI
  - sarchertech 3y ago
    If you only use one of those, for everything it’s easier. I still wouldn’t say it’s easy because it requires thought, and in my experience is commonly done incorrectly. I’ve regularly seen anything from picking the wrong partition key, to grabbing a bunch of messages from a single partition to process concurrently. If you have an existing system where some services are fully event sourced, or some depend on complicated database transactions but still want to publish to a Kafka topic etc…, it gets more complicated.
tasn 3y ago
Ordering is even more compelx than that! I previously wrote a post about it as it comes up a lot with Svix. https://www.svix.com/blog/guaranteeing-webhook-ordering/ https://www.svix.com/blog/guaranteeing-webhook-ordering/
Semaphor 3y ago
I've only ever worked with two webhook systems (Eventbrite and PayPal) and both have those issues handled. I'd say you should just look how others solved it.

sarchertech 3y ago

Ordering is something that everyone always forgets. The only way to actually guarantee they are sent and received in order is to use a lock on whatever subset of messages you care about being ordered. Then only process one of those at a time. At my last company I built an ordered transaction outbox to handle this. Each message has a partition key. When the worker picks up a new message to process it locks the partition key, so that it’s the only worker processing any messages with that partition key. Different messages use different things for their partition key. For some messages where we only care about the order within a given order (it’s an order management system) we use the order_id. For other messages where we care about global ordering we use the tenant_id etc… You could just use tenant_id for every message, but by using the most granular partition that you can, you can get a lot more parallelism.

foobiekr 3y ago
Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases. You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model). That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier. The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing. Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.
- Toutouxc 3y ago
  I would really like to understand the second paragraph, but I don’t understand a single sentence. What exactly are you proposing? What’s a state compressed reporter?
  - 15457345234 3y ago
    Their comment makes no real sense to me either. I can't tell from looking at their comment history whether or not they're AI, but their profile says 'I am nobody.' It's a strange world.
  - foobiekr 3y ago
    The reporter is the system that sends updates to your external observers; observers register with the reporter and the reporter facilitates their consumption of events. In the case of a webhook, this is the first tier (the actual event store) that individual webhooks contexts are following (on the service side). You need these to be per-outbound-destination to avoid head of line blocking. I am proposing that people use state synchronization paradigms and not simple event replication for their notification systems. Since the remote side is not within your control (typically) the state compression all needs to be service side. If you want to explore systems like this, you should start with the basics of how routing protocol peering actually works inside routing protocol implementations. In that context, you have dozens to hundreds of fast and slow consumers that are trying to converge with what is, basically a forever changing data set through the exchange of adds, deletes, and updates with peers. Any given downstream peer may block for extended periods of time, they may resume, and so on, and you cannot buffer significant amounts of data per peer (since the stack _must_ consume updates from its own peers). Alternately, start looking into how databases sync with one another beyond the simple replicated transaction log sense.
- sarchertech 3y ago
  In many cases none of that is possible because you aren’t designing the entire system. Either because it’s already been partially implemented, or because you’re modeling a system that does require ordering with no drops. And whether webhooks are a good choice or not, they are often required for one reason or another.
dalore 3y ago
Could you not timestamp your message and then let the client do the ordering. Granted they might get messages out of order, but I never promised them their webhooks would be in order.
- devoutsalsa 3y ago
  Regular timestamps can only be used reliably for ordering if all the timestamps come from a single system. Different systems have different clocks, so they'll never be precisely in sync. You can also have timestamp collisions, which makes ordering ambiguous.
  - chrsig 3y ago
    This is assuming the use of a monotonic clock. the system wall clock may jump backwards in time. Itt also assumes that only one id can be generated per millisecond (or whatever clock resolution). In the case of multiple threads or processes on multiple cores, it becomes more likely that there will be a collision. Timestamps don't make for a great ordering key.
- paulddraper 3y ago
  Yes, you can push the problem to your users.
- skeeter2020 3y ago
  Timestamps from where? There is not necessarily going to be a central authority or processor who can guarantee these are in sync across distributed systems
paulddraper 3y ago
> At my last company I built an ordered transaction outbox to handle this. Almost all message queue/event streaming systems support FIFO (SQS, RabbitMQ, ActiveMQ, Kafka, Kinesis). If you use one of these, it's easy to solve ordering.
- skeeter2020 3y ago
  This is just for arrival order though. The GP mentions transaction-based operations; handling this without massive hits to parallel processing is not an easy problem to solve.
  - paulddraper 3y ago
    ? I'm not sure I understand. You choose an FIFO key, e.g. order ID. Any message with that key is processed serially. Ordering between keys is unspecified. It is important to choose a key that guarantees sufficient ordering (for correctness), but not too much (for performance).
    - ignoramous 3y ago
      > You choose an FIFO key, e.g. order ID. This is the part of the system that is doing most of the heavy lifting. Ex A, The struggles of Elasticsearch: https://archive.is/rm1UI https://archive.is/rm1UI
- sarchertech 3y ago
  If you only use one of those, for everything it’s easier. I still wouldn’t say it’s easy because it requires thought, and in my experience is commonly done incorrectly. I’ve regularly seen anything from picking the wrong partition key, to grabbing a bunch of messages from a single partition to process concurrently. If you have an existing system where some services are fully event sourced, or some depend on complicated database transactions but still want to publish to a Kafka topic etc…, it gets more complicated.

tasn 3y ago

Ordering is even more compelx than that! I previously wrote a post about it as it comes up a lot with Svix. https://www.svix.com/blog/guaranteeing-webhook-ordering/ https://www.svix.com/blog/guaranteeing-webhook-ordering/

Semaphor 3y ago

I've only ever worked with two webhook systems (Eventbrite and PayPal) and both have those issues handled. I'd say you should just look how others solved it.