6 ms·

Webhook Failure Scenarios

100 points by luu 3y ago

leetrout 3y ago
That needs better naming in my opinion. Just sender and receiver would be enough IMO. Maybe it is just me but it is unusual to see origin used in this context even if it is technically correct I only ever see it used with CDNs and load balancing.
- benburleson 3y ago
  I’ve always referred to those client/server.
  - leetrout 3y ago
    I think webhooks are even more misleading because they are server to server.
    - janci 3y ago
      And the client is actually the sender.
- hermanradtke 3y ago
  I was frustrated with my naming too. > technically correct I only ever see it used with CDNs and load balancing That is why I used it! I responded in https://news.ycombinator.com/item?id=37525971 https://news.ycombinator.com/item?id=37525971 that maybe I should change it. Sender/receiver is better.
JimDabell 3y ago
Some interesting previous discussions here: - Best Practices for Using Webhooks (stripe.com): https://news.ycombinator.com/item?id=32521159 https://news.ycombinator.com/item?id=32521159 - Collection of best practices for providing and consuming webhooks (webhooks.fyi): https://news.ycombinator.com/item?id=32517645 https://news.ycombinator.com/item?id=32517645 - Give me /events, not webhooks (sequin.io): https://news.ycombinator.com/item?id=27823109 https://news.ycombinator.com/item?id=27823109
losvedir 3y ago
Webhooks are fun to think about. A couple more issues off the top of my head: * Ordering. Since network requests can take variable amounts of time, how do you ensure those two "foo.updated" events are processed in order, or that the receiver can tell their order? Especially something to consider if the webhooks will retry a few times on intermittent failures. * Consistency. Always a concern in distributed microservice land, but maybe more acute when generating webhooks right as things are updated: if the receiver uses the webhook to make an API request right back into the system, will the API have the same view of the data? * DDoS. How do you make sure the webhook destination URL is owned by the subscriber? If your system can generate high volumes of webhook traffic, that could be a problem. * Infinite loops. A silly one, but the user could conceivably point the webhook at a URL of the system that sends the webhook, in such a way that it will cause a new webhookable event to be generated.
- sarchertech 3y ago
  Ordering is something that everyone always forgets. The only way to actually guarantee they are sent and received in order is to use a lock on whatever subset of messages you care about being ordered. Then only process one of those at a time. At my last company I built an ordered transaction outbox to handle this. Each message has a partition key. When the worker picks up a new message to process it locks the partition key, so that it’s the only worker processing any messages with that partition key. Different messages use different things for their partition key. For some messages where we only care about the order within a given order (it’s an order management system) we use the order_id. For other messages where we care about global ordering we use the tenant_id etc… You could just use tenant_id for every message, but by using the most granular partition that you can, you can get a lot more parallelism.
  - foobiekr 3y ago
    Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases. You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model). That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier. The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing. Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.
sergiotapia 3y ago
At the most basic level, what I like to do is create a webhook response controller, the controller enqueues a job and immediately returns an HTTP 200. The job is saved in my postgres database, using Oban (Elixir) so I have observability, accurate bug stacktraces are saved in the job record's error field, and I can see how many times the job has been attempted.
naikrovek 3y ago
right out of the gate: why not use the terms "sender" and "receiver" instead of "client" and "origin"? the author uses "client" to refer to the application sending the webhook, and "origin" to refer to the application receiving the webhook. this is all backwards to me. I often say that naming things is hard, but it isn't so hard that this needs be the result.
- hermanradtke 3y ago
  Fair call out. I responded to similar feedback here: https://news.ycombinator.com/item?id=37525992 https://news.ycombinator.com/item?id=37525992
- JuanPosadas 3y ago
  FWIW I wrote a program that used webhooks and RSS feeds and needed to prefix both with their direction, I ended up choosing inbound/outbound.
tasn 3y ago
There are so many more failure scenarios! I wrote a post about common TLS errors recently: https://www.svix.com/blog/ssl-tls-incomplete-certificate-chain/ https://www.svix.com/blog/ssl-tls-incomplete-certificate-cha...
nonane 3y ago
Any suggestions to alternative to webhooks? I feel a ‘pull’ based model with cursors and long polling would be simpler and more reliable than webhooks.
- andenacitelli 3y ago
  Webhooks tend to make a lot more sense in event-driven applications . They introduce additional complexity when it comes to all sorts of edge cases - agree that a pull system is probably much easier to do error handling for, my only concern would probably be performance.
- 10000truths 3y ago
  WebSockets.
- JuanPosadas 3y ago
  Simple: cron job, one per minute, check for recently created (and/or updated) entries.
pyrolistical 3y ago
The way I solve all these problems is. Prerequisite: require receiver to be able to process duplicate webhook being resent. Algorithm outline: 1. Have webhook body stored by sender 2. Attempt to send body to receiver 3. If 2x status returned mark webhook as sent. Done 4. Otherwise for any other status or timeout increment attempt for webhook and try again later until maximum attempts (which then mark webhook as failed) This covers the tricky case where success response gets dropped in network by having the sender time out and retry later. The receiver gets a duplicate
- foobiekr 3y ago
  You should describe how you are backpressuring internally while this is going on and also how you send an indicator to the receiver that they encountered a gap.
bennyp101 3y ago
Those terms are very confusing - to me orign would be the SENDER ... where the message ORIGNates from ... But yes, those are common scenarios, and most services that offer calling a webhook have retries baked in for such things. It is the responsibility of the consumer to verify if they have already received a message or not. You should not be consuming a webhook 'live' and attempting to work on it the moment you receive it - that way lies dragons! I have dedicated endpoints for services to send to, and they are verified and consumed as quickly as possible and put into a queue to be processed after. That way I can at least see what was sent, and if we missed something.
- hermanradtke 3y ago
  > to me orign would be the SENDER My history with CDNs is that origin refers to the server sitting behind the proxy. I do agree that > where the message ORIGNates from makes sense too.
  - paulddraper 3y ago
    A CDN origin refers to the server where the content ORIGINates from.
chunk_waffle 3y ago
For a server sending webhooks to endpoints entered in by users, take care that: a: The FQDN does not resolve to an RFC1918 address (You don't want to be POSTing payloads to endpoints within your internal network.) b: If you respond to redirect responses (easier to just not do so for other reasons as well) also make sure those don't resolve to internal addresses too!
- foobiekr 3y ago
  The attacker-controlled DNS record pointing to an internal private address or an explicit redirect is a classic, especially if they can control the event template being used and the service relies entirely on edge filtering... Too much template control is a risk. I mean, there's a lot of things you should do when dealing with this that most people don't pay attention to: https://datatracker.ietf.org/doc/html/rfc2606 https://datatracker.ietf.org/doc/html/rfc2606 https://datatracker.ietf.org/doc/html/rfc3927 https://datatracker.ietf.org/doc/html/rfc3927 https://datatracker.ietf.org/doc/html/rfc4193 https://datatracker.ietf.org/doc/html/rfc4193 https://datatracker.ietf.org/doc/html/rfc6761 https://datatracker.ietf.org/doc/html/rfc6761 ... and so on. At least in Go some of the handy checks are simplified by IP.Is(Private|Loopback|Multicast|InterfaceLocalMulticast()|LinkLocal*etc.)
- nicolaslem 3y ago
  It's been years since I've looked into this problem but to tackle it properly one shouldn't just resolve the domain and check that the IP is acceptable. The HTTP client library needs to be involved by providing a way to run code just before creating the socket, which very few do.
  - chunk_waffle 3y ago
    Oh interesting... would love to hear more about that. I guess what could happen is: You would: 1. Resolve the FQDN 2. check the IP 3. Make the request When the request is actually made, the FQDN is resolved again and a different IP is returned?
caseysoftware 3y ago
Very cool article. I'm one of the creators of https://webhooks.fyi/ https://webhooks.fyi/ and will do a PR to add this one. Thanks!
- andenacitelli 3y ago
  Cool site. Seems like a great learning resource, nice work. (No affiliation, just a sucker for good docs)

100 points by luu 3y ago

That needs better naming in my opinion. Just sender and receiver would be enough IMO. Maybe it is just me but it is unusual to see origin used in this context even if it is technically correct I only ever see it used with CDNs and load balancing.

benburleson 3y ago
I’ve always referred to those client/server.
- leetrout 3y ago
  I think webhooks are even more misleading because they are server to server.
  - janci 3y ago
    And the client is actually the sender.
hermanradtke 3y ago
I was frustrated with my naming too. > technically correct I only ever see it used with CDNs and load balancing That is why I used it! I responded in https://news.ycombinator.com/item?id=37525971 https://news.ycombinator.com/item?id=37525971 that maybe I should change it. Sender/receiver is better.

JimDabell 3y ago

Some interesting previous discussions here: - Best Practices for Using Webhooks (stripe.com): https://news.ycombinator.com/item?id=32521159 https://news.ycombinator.com/item?id=32521159 - Collection of best practices for providing and consuming webhooks (webhooks.fyi): https://news.ycombinator.com/item?id=32517645 https://news.ycombinator.com/item?id=32517645 - Give me /events, not webhooks (sequin.io): https://news.ycombinator.com/item?id=27823109 https://news.ycombinator.com/item?id=27823109

losvedir 3y ago

Webhooks are fun to think about. A couple more issues off the top of my head: * Ordering. Since network requests can take variable amounts of time, how do you ensure those two "foo.updated" events are processed in order, or that the receiver can tell their order? Especially something to consider if the webhooks will retry a few times on intermittent failures. * Consistency. Always a concern in distributed microservice land, but maybe more acute when generating webhooks right as things are updated: if the receiver uses the webhook to make an API request right back into the system, will the API have the same view of the data? * DDoS. How do you make sure the webhook destination URL is owned by the subscriber? If your system can generate high volumes of webhook traffic, that could be a problem. * Infinite loops. A silly one, but the user could conceivably point the webhook at a URL of the system that sends the webhook, in such a way that it will cause a new webhookable event to be generated.

sarchertech 3y ago
Ordering is something that everyone always forgets. The only way to actually guarantee they are sent and received in order is to use a lock on whatever subset of messages you care about being ordered. Then only process one of those at a time. At my last company I built an ordered transaction outbox to handle this. Each message has a partition key. When the worker picks up a new message to process it locks the partition key, so that it’s the only worker processing any messages with that partition key. Different messages use different things for their partition key. For some messages where we only care about the order within a given order (it’s an order management system) we use the order_id. For other messages where we care about global ordering we use the tenant_id etc… You could just use tenant_id for every message, but by using the most granular partition that you can, you can get a lot more parallelism.
- foobiekr 3y ago
  Well for starters, you should by-design not create an interface where individual discrete events are required to be delivered, without drops or strictly in order. Once you start down that path, you have to be able to answer what you do when the the destination is blocked (for any reason: network congestion or outage, service outage, etc.). This is the path where your internal queue (whether physical or virtual - that is, a cursor) starts requiring potential days of retention and the ability to stream from any point in that history. Been there, done that. You can implement that retention in all sorts of ways (rows in a database, messages in kafka, etc.) each of which has interesting cost dynamics and corner cases. You are almost better off with a state compressed reporter that has the semantics of always reporting a _sensible_ series of events to the external endpoint where _dependent_ events have sensible relative order ("x was added, x was deleted") that converges on current state no matter how slow the outbound endpoint is and without regard to how many of them there are. No key locking and no buffering, O(1) storage per walker - just a walk on conveying state. This requires careful design of both the system _and_ the schema (boolean state needs to be accompanied with a counter, for example, to convey collapsed/consolidated history of the state changes in this model). That's a lot more work, but it is possible, this kind of thing dates to the 80s or earlier. The only in-order/no-drops case that is really common is logs (which is a poor way of conveying event/state tracking) and webhooks are a very poor choice for that kind of thing. Also, in all cases, receivers should have to accept that they may receive the same notification more than once. Obviosuly there are ways around this, but they are expensive at scale.

sergiotapia 3y ago

At the most basic level, what I like to do is create a webhook response controller, the controller enqueues a job and immediately returns an HTTP 200. The job is saved in my postgres database, using Oban (Elixir) so I have observability, accurate bug stacktraces are saved in the job record's error field, and I can see how many times the job has been attempted.

naikrovek 3y ago

right out of the gate: why not use the terms "sender" and "receiver" instead of "client" and "origin"? the author uses "client" to refer to the application sending the webhook, and "origin" to refer to the application receiving the webhook. this is all backwards to me. I often say that naming things is hard, but it isn't so hard that this needs be the result.

hermanradtke 3y ago
Fair call out. I responded to similar feedback here: https://news.ycombinator.com/item?id=37525992 https://news.ycombinator.com/item?id=37525992
JuanPosadas 3y ago
FWIW I wrote a program that used webhooks and RSS feeds and needed to prefix both with their direction, I ended up choosing inbound/outbound.

tasn 3y ago

There are so many more failure scenarios! I wrote a post about common TLS errors recently: https://www.svix.com/blog/ssl-tls-incomplete-certificate-chain/ https://www.svix.com/blog/ssl-tls-incomplete-certificate-cha...

nonane 3y ago

Any suggestions to alternative to webhooks? I feel a ‘pull’ based model with cursors and long polling would be simpler and more reliable than webhooks.

andenacitelli 3y ago
Webhooks tend to make a lot more sense in event-driven applications . They introduce additional complexity when it comes to all sorts of edge cases - agree that a pull system is probably much easier to do error handling for, my only concern would probably be performance.
10000truths 3y ago
WebSockets.
JuanPosadas 3y ago
Simple: cron job, one per minute, check for recently created (and/or updated) entries.

pyrolistical 3y ago

The way I solve all these problems is. Prerequisite: require receiver to be able to process duplicate webhook being resent. Algorithm outline: 1. Have webhook body stored by sender 2. Attempt to send body to receiver 3. If 2x status returned mark webhook as sent. Done 4. Otherwise for any other status or timeout increment attempt for webhook and try again later until maximum attempts (which then mark webhook as failed) This covers the tricky case where success response gets dropped in network by having the sender time out and retry later. The receiver gets a duplicate

foobiekr 3y ago
You should describe how you are backpressuring internally while this is going on and also how you send an indicator to the receiver that they encountered a gap.

bennyp101 3y ago

Those terms are very confusing - to me orign would be the SENDER ... where the message ORIGNates from ... But yes, those are common scenarios, and most services that offer calling a webhook have retries baked in for such things. It is the responsibility of the consumer to verify if they have already received a message or not. You should not be consuming a webhook 'live' and attempting to work on it the moment you receive it - that way lies dragons! I have dedicated endpoints for services to send to, and they are verified and consumed as quickly as possible and put into a queue to be processed after. That way I can at least see what was sent, and if we missed something.

hermanradtke 3y ago
> to me orign would be the SENDER My history with CDNs is that origin refers to the server sitting behind the proxy. I do agree that > where the message ORIGNates from makes sense too.
- paulddraper 3y ago
  A CDN origin refers to the server where the content ORIGINates from.

chunk_waffle 3y ago

For a server sending webhooks to endpoints entered in by users, take care that: a: The FQDN does not resolve to an RFC1918 address (You don't want to be POSTing payloads to endpoints within your internal network.) b: If you respond to redirect responses (easier to just not do so for other reasons as well) also make sure those don't resolve to internal addresses too!

foobiekr 3y ago
The attacker-controlled DNS record pointing to an internal private address or an explicit redirect is a classic, especially if they can control the event template being used and the service relies entirely on edge filtering... Too much template control is a risk. I mean, there's a lot of things you should do when dealing with this that most people don't pay attention to: https://datatracker.ietf.org/doc/html/rfc2606 https://datatracker.ietf.org/doc/html/rfc2606 https://datatracker.ietf.org/doc/html/rfc3927 https://datatracker.ietf.org/doc/html/rfc3927 https://datatracker.ietf.org/doc/html/rfc4193 https://datatracker.ietf.org/doc/html/rfc4193 https://datatracker.ietf.org/doc/html/rfc6761 https://datatracker.ietf.org/doc/html/rfc6761 ... and so on. At least in Go some of the handy checks are simplified by IP.Is(Private|Loopback|Multicast|InterfaceLocalMulticast()|LinkLocal*etc.)
nicolaslem 3y ago
It's been years since I've looked into this problem but to tackle it properly one shouldn't just resolve the domain and check that the IP is acceptable. The HTTP client library needs to be involved by providing a way to run code just before creating the socket, which very few do.
- chunk_waffle 3y ago
  Oh interesting... would love to hear more about that. I guess what could happen is: You would: 1. Resolve the FQDN 2. check the IP 3. Make the request When the request is actually made, the FQDN is resolved again and a different IP is returned?

caseysoftware 3y ago

Very cool article. I'm one of the creators of https://webhooks.fyi/ https://webhooks.fyi/ and will do a PR to add this one. Thanks!

andenacitelli 3y ago
Cool site. Seems like a great learning resource, nice work. (No affiliation, just a sucker for good docs)