4 ms·
This is a hill I find myself frequently fighting (and losing on): webhooks are terrible to maintain, because they start from the premise "this never breaks" and
by XorNot 5y ago
This is a hill I find myself frequently fighting (and losing on): webhooks are terrible to maintain, because they start from the premise "this never breaks" and thats about where development in an organization stops.
The only event API I ever want is notifications there's new data, and then an interface by which I can query all new data which has arrived by some sort of index marker - because this is fundamentally reliable. It means whatever happens to my system, I can reliably recover missed events, skipped events, or rebuild from previous events.
And this is in fact exactly how something like Kafka actually works! Complete with first-class support for compacting queues to produce valid "summarized" starting points.
Any streaming system essentially should never start as a streaming system - it should start as a slow-path pull-based system, and have a fast-path push system added on top of it if needed - because then you've built your recovery path already, rather then what happens way too often which is just "oh yeah, we'll develop that when it breaks".
- sroussey 5y agoI agree. A simple ping with the latest ID (which is option for you to use to get events from last ID to newest ID). The go get the events, which is a likely reusing code. Polling is crap. Extra points for being able to set something like 1s between pings (now you see why I like the option ID for a range).
- delusional 5y ago> Any streaming system essentially should never start as a streaming system - it should start as a slow-path pull-based system, and have a fast-path push system added on top of it if needed - because then you've built your recovery path already, rather then what happens way too often which is just "oh yeah, we'll develop that when it breaks". I think this is a quite interesting and important point. When we talk about "doing the simple thing first" too often we end building something that is technically simple but fickle. The trick to making the simple thing reliable is to figure out which part is the slow-path (or failure mode), and then only building that. Unfortunately, it often means out result ends up technically "boring" since all the interesting optimizations are what we cut out, but I think that's worth it if the end result is a more useful product. It's something I've been working with and thinking about for a while. I think it applies to a way broader scope than this discussion.
- rattray 5y ago(I worked on the same team as bkrausz, elsewhere on this thread, albeit not concurrently). Yes, this is pretty much the right thing to do. It can be a bit more work for the API consumer, partly because they need to track state of their last-read ID, and there's more moving parts. If you're building a webhhook+events system like Stripe's, you might consider adding an option for a mostly-empty webhook body, which can speed things up in this use-case, but still allows "the easy way" of just processing the event from within the webhook body. (For readers thinking of implementing this, note that "query for new data" means hitting a dedicated /events api, not individual tables, which might have unpleasant load/performance consequences).