4 ms·

I read about a strategy of having Webhooks, and supplementing with polling an "/events" endpoint that has a historical record of events for occasion backfilling

by Msw242 4y ago

I read about a strategy of having Webhooks, and supplementing with polling an "/events" endpoint that has a historical record of events for occasion backfilling.

tasn 4y ago
Yup, that's one way of doing it (which we support at Svix). Though another great way is "thin webhooks", where you just use the webhook to notify your customers something happened, and then they can use your API to get the exact data that changed.
- deleted 4y ago
  [deleted]
- Canada 4y ago
  I'm annoyed by these. Just give me the payload, authenticated of course. I can always just ignore your payload/signature and treat it as a notification to poll you anyway.
  - tasn 4y ago
    Yeah, but you need to know what changed, so you need the relevant IDs.
    - Canada 4y ago
      Yeah, so give em to me in the event endpoint cause for sure you have them. I will iterate it and do what I need to do. The key is these things need to have IDs and be idempotent always. Do that, and getting out of a lot of big screw ups needs no further effort/thinking after a reboot or rollback.
RSZC 4y ago
The WebSub spec recommends this approach, and I've implemented it, and I really would recommend against doing this. It adds a tremendous amount of complexity to all of your services, where now all of your services will need to maintain the ability to answer the question 'what was your state at this point in time.' It makes it extremely difficult to support new webhooks, and honestly in my experience nobody actually ever queries for the historical data anyway. My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. They're a great addition for stuff outside the critical path where delivery is not essential, and as long as you stick to that paradigm you can dodge these issues entirely. Re 'thin webhooks' which require hydration: - subject to either complexity to support historical state (e.g. what was the state at the point in time when the webhook was issued) or race conditions (two things happened, but your API only gets current state) - honestly none of your consumers want this - they're gonna be like 'why do i have to also query an API why can't you just hydrate the payload in the webhook body'. Guaranteed. - still probably the correct solution if the data is super private and you want to be extremely cautious about issuing webhooks for it source: worked exclusively on webhooks and similar eventing stuffs at place you've heard of for a few years
- twblalock 4y ago
  Yep, I've seen all of this play out too. IMO webhooks are best used when your SLA for delivery is "best effort." Anytime you need to guarantee delivery (including guaranteeing the delivered data was correctly processed by the receiver), or guarantee anything about ordering or replayability, they just aren't that great.
- lliamander 4y ago
  > My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. What is your recommendation for situations that are absolutely critical? Is the recommendation to just have an /events endpoint? I'm honestly curious because I have also implemented several wehbook-based systems over the past few years, and definitely seen the downsides, and so I'm trying to figure out the best way to manage asynchronous processing in distributed systems.
- Canada 4y ago
  > It adds a tremendous amount of complexity to all of your services But it makes my life so much easier as a user of your service. I admit that it's not worth it for every type of service, but for financial ones it's 100% worth it. I'm trying to keep my ledger balances in sync with yours. I hate having to make a manual adjustment because something got missed. Having my system heal itself even in the case when we really fucked up, pushed that bad update where we crashed and didn't persist what you said, or cloudflare fucked us and blocked you, or something... there's always something... I really really appreciate that. I mean, if you don't because it's too complex for you.. okay, I'll have have to add the complexity on my side to compensate. I'm really thankful when I don't have to, and when I'm the event/truth source for others I try my best to do that courtesy for the event consumers.

by Msw242 4y ago

I read about a strategy of having Webhooks, and supplementing with polling an "/events" endpoint that has a historical record of events for occasion backfilling.

tasn 4y ago

Yup, that's one way of doing it (which we support at Svix). Though another great way is "thin webhooks", where you just use the webhook to notify your customers something happened, and then they can use your API to get the exact data that changed.

deleted 4y ago
[deleted]
Canada 4y ago
I'm annoyed by these. Just give me the payload, authenticated of course. I can always just ignore your payload/signature and treat it as a notification to poll you anyway.
- tasn 4y ago
  Yeah, but you need to know what changed, so you need the relevant IDs.
  - Canada 4y ago
    Yeah, so give em to me in the event endpoint cause for sure you have them. I will iterate it and do what I need to do. The key is these things need to have IDs and be idempotent always. Do that, and getting out of a lot of big screw ups needs no further effort/thinking after a reboot or rollback.

RSZC 4y ago

The WebSub spec recommends this approach, and I've implemented it, and I really would recommend against doing this. It adds a tremendous amount of complexity to all of your services, where now all of your services will need to maintain the ability to answer the question 'what was your state at this point in time.' It makes it extremely difficult to support new webhooks, and honestly in my experience nobody actually ever queries for the historical data anyway. My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. They're a great addition for stuff outside the critical path where delivery is not essential, and as long as you stick to that paradigm you can dodge these issues entirely. Re 'thin webhooks' which require hydration: - subject to either complexity to support historical state (e.g. what was the state at the point in time when the webhook was issued) or race conditions (two things happened, but your API only gets current state) - honestly none of your consumers want this - they're gonna be like 'why do i have to also query an API why can't you just hydrate the payload in the webhook body'. Guaranteed. - still probably the correct solution if the data is super private and you want to be extremely cautious about issuing webhooks for it source: worked exclusively on webhooks and similar eventing stuffs at place you've heard of for a few years

twblalock 4y ago
Yep, I've seen all of this play out too. IMO webhooks are best used when your SLA for delivery is "best effort." Anytime you need to guarantee delivery (including guaranteeing the delivered data was correctly processed by the receiver), or guarantee anything about ordering or replayability, they just aren't that great.
lliamander 4y ago
> My opinion is that webhooks just aren't the correct approach to take for anything absolutely critical. What is your recommendation for situations that are absolutely critical? Is the recommendation to just have an /events endpoint? I'm honestly curious because I have also implemented several wehbook-based systems over the past few years, and definitely seen the downsides, and so I'm trying to figure out the best way to manage asynchronous processing in distributed systems.
Canada 4y ago
> It adds a tremendous amount of complexity to all of your services But it makes my life so much easier as a user of your service. I admit that it's not worth it for every type of service, but for financial ones it's 100% worth it. I'm trying to keep my ledger balances in sync with yours. I hate having to make a manual adjustment because something got missed. Having my system heal itself even in the case when we really fucked up, pushed that bad update where we crashed and didn't persist what you said, or cloudflare fucked us and blocked you, or something... there's always something... I really really appreciate that. I mean, if you don't because it's too complex for you.. okay, I'll have have to add the complexity on my side to compensate. I'm really thankful when I don't have to, and when I'm the event/truth source for others I try my best to do that courtesy for the event consumers.