3 ms·

Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in orde

Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases. You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model). That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier. The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing. Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.

Toutouxc 3y ago
I would really like to understand the second paragraph, but I don’t understand a single sentence. What exactly are you proposing? What’s a state compressed reporter?
- 15457345234 3y ago
  Their comment makes no real sense to me either. I can't tell from looking at their comment history whether or not they're AI, but their profile says 'I am nobody.' It's a strange world.
- foobiekr 3y ago
  The reporter is the system that sends updates to your external observers; observers register with the reporter and the reporter facilitates their consumption of events. In the case of a webhook, this is the first tier (the actual event store) that individual webhooks contexts are following (on the service side). You need these to be per-outbound-destination to avoid head of line blocking. I am proposing that people use state synchronization paradigms and not simple event replication for their notification systems. Since the remote side is not within your control (typically) the state compression all needs to be service side. If you want to explore systems like this, you should start with the basics of how routing protocol peering actually works inside routing protocol implementations. In that context, you have dozens to hundreds of fast and slow consumers that are trying to converge with what is, basically a forever changing data set through the exchange of adds, deletes, and updates with peers. Any given downstream peer may block for extended periods of time, they may resume, and so on, and you cannot buffer significant amounts of data per peer (since the stack _must_ consume updates from its own peers). Alternately, start looking into how databases sync with one another beyond the simple replicated transaction log sense.
sarchertech 3y ago
In many cases none of that is possible because you aren’t designing the entire system. Either because it’s already been partially implemented, or because you’re modeling a system that does require ordering with no drops. And whether webhooks are a good choice or not, they are often required for one reason or another.

Toutouxc 3y ago

I would really like to understand the second paragraph, but I don’t understand a single sentence. What exactly are you proposing? What’s a state compressed reporter?

15457345234 3y ago
Their comment makes no real sense to me either. I can't tell from looking at their comment history whether or not they're AI, but their profile says 'I am nobody.' It's a strange world.
foobiekr 3y ago
The reporter is the system that sends updates to your external observers; observers register with the reporter and the reporter facilitates their consumption of events. In the case of a webhook, this is the first tier (the actual event store) that individual webhooks contexts are following (on the service side). You need these to be per-outbound-destination to avoid head of line blocking. I am proposing that people use state synchronization paradigms and not simple event replication for their notification systems. Since the remote side is not within your control (typically) the state compression all needs to be service side. If you want to explore systems like this, you should start with the basics of how routing protocol peering actually works inside routing protocol implementations. In that context, you have dozens to hundreds of fast and slow consumers that are trying to converge with what is, basically a forever changing data set through the exchange of adds, deletes, and updates with peers. Any given downstream peer may block for extended periods of time, they may resume, and so on, and you cannot buffer significant amounts of data per peer (since the stack _must_ consume updates from its own peers). Alternately, start looking into how databases sync with one another beyond the simple replicated transaction log sense.

sarchertech 3y ago

In many cases none of that is possible because you aren’t designing the entire system. Either because it’s already been partially implemented, or because you’re modeling a system that does require ordering with no drops. And whether webhooks are a good choice or not, they are often required for one reason or another.