6 ms·

Collection of best practices for providing and consuming webhooks

235 points by m3h 4y ago

Traubenfuchs 4y ago
Why are callback registration and async APIs now called webhooks?
- deleted 4y ago
  [deleted]
- osullip 4y ago
  Not sure of the evolution, but maybe because 'callbacks' exist everywhere, but 'webhooks' are specific to REST APIs.
- pavlov 4y ago
  The usual nomenclature is that regular callbacks and async API calls are made and registered in the same process space as the caller. (E.g. passing a function pointer in C, or using “await” in modern languages.) Webhooks are callbacks received over HTTP from a public API. They may not be programmatically registered at all, e.g. the callback URL may be configured manually in an admin dashboard. So the application has less control over webhooks vs. regular callbacks or async functions.
cuu508 4y ago
There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be able to deal with that. * Timeouts (slowloris): the webhook target could be sending back one byte at a time, with 1 second pauses inbetween. If you are using, say, the "requests" python library for making HTTP requests, the "timeout" parameter will not help here * Private IPs and reserved IPs: you probably don't want users defining webhooks to http://127.0.0.1:<some-port> http://127.0.0.1:<some-port> and probing your internal network. Remember about private IPv6 ranges too * Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP. It is not enough to just validate webhook URLs when users set them up. * HTTP redirects to private IPs. If your HTTP client library follows HTTP redirects, the attacker can set up a webhook endpoint that redirects to a private IP. Again, it is not enough to validate the user-supplied URL. * Excessive HTTP redirects. The attacker can set up a redirect loop - make sure this does not circumvent your timeout setting. My current solution for all of the above is to use libcurl via pycurl. I wrote a wrapper that mimics requests API: https://github.com/healthchecks/healthchecks/blob/master/hc/lib/curl.py https://github.com/healthchecks/healthchecks/blob/master/hc/... (may contain bugs, use at your own risk :-)
- technion 4y ago
  Just adding to that - don't forget about users defining AWS metadata addresses for a webhook. Returning IAM data to them can be .. bad.
- m3h 4y ago
  It seems like webhooks have enough corner cases for the sender to require a specialized tool to protect itself from malicious users and to stay performant. Does anyone have suggestion for such tools/services that they might have used in production?
  - calyhre 4y ago
    At Slite, for all outgoing calls we use a sandboxed proxy. It has saved us a few times already. We detailed the trick in a blog post -> https://slite-tech-blog.ghost.io/anti-ssrf-solution/ https://slite-tech-blog.ghost.io/anti-ssrf-solution/
    - freeqaz 4y ago
      This is the path I've seen be fairly robust at a few tech companies I've helped sort out this defense for. I've helped write libraries too but the proxy is the easiest approach when targeting many languages.
  - charcircuit 4y ago
    For half of those it's called a firewall and for the other half you should specify timeouts / max redirects.
  - rajamaka 4y ago
    Outgoing webhooks dispatched from a Lambda seemed to solve most of the above problems for us
    - tasn 4y ago
      Using lambda doesn't protect you against any of these. Timeout attacks: instead of failing you'll now be just be paying through the nose. And also with enough scale you can hit lambda limits and fail. SSRF: you are still as vulnerable unless the lambda is outside your VPC (but then it's the VPC that solved it, not the lambda).
sparsely 4y ago
As this page makes very clear, it's actually pretty hard to make a robust webhooks implementation! What alternatives are there? I've looked at: * Publishing AWS EventBridge events to other accounts. * /events instead of webhooks https://blog.sequin.io/events-not-webhooks/ https://blog.sequin.io/events-not-webhooks/ * ???
- floucky 4y ago
  Emails maybe ? But it means you have to regularly check your inbox. Or use a webhook called by your email provider when you receive an email :D
- tasn 4y ago
  We built Svix[1] exactly for this reason. We make it super easy for companies to send webhooks reliably. There are only three companies with a "perfect score" on webhooks.fyi's list, we power one of them. ;) [1] https://www.svix.com https://www.svix.com
  - caseysoftware 4y ago
    That's awesome and glad to see others making it easier and better to do this right. Obviously you should add some of your other customers to make sure they get the attention they deserve. Also, there are other patterns or practices that should be in the list, let us know. PRs are welcome but feel free to email me directly: danger@ngrok.com
    - tasn 4y ago
      Will email you now in a bit! I also just replied to your other comment about another topic, so there's much to discuss. :)
- Traubenfuchs 4y ago
  > What alternatives are there? At the top of my head: Short Polling, long polling, websockets, response streaming, SSE, SNS, paging API, comet...
egorfine 4y ago
Excellent. I'm in process of building a service delivering notifications via webhooks right now. Thank you!
- tasn 4y ago
  Shameless plug: we make it super easy for companies to send webhooks. https://www.svix.com https://www.svix.com There are only 3 companies on webhooks.fyi's list with 3 green checks, and we're powering webhooks for one of those. :)
  - egorfine 4y ago
    Thank you! Looks promising!
danpalmer 4y ago
Great resource! I’ve encountered most of these problems at some point. It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.
- tasn 4y ago
  I recorded a video with advice about designing a webhooks system a couple of months ago: https://www.svix.com/blog/webhook-architecture-design/ https://www.svix.com/blog/webhook-architecture-design/ It's high level, so it doesn't touch everything you mentioned, but it's a good primer.
caseysoftware 4y ago
Hello all! I'm one of the creators of webhooks.fyi over at https://ngrok.com/ https://ngrok.com/ Happy to answer any questions! First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole. Next, yes it's ridiculously hard to build webhooks correctly. There are so many shortcuts that feel "okay" and you don't really think about until you realize that it didn't quite work as you had planned and now you have a gap. We're hoping the site can help people move more of that thinking earlier and make better decisions. Finally, we're missing a bunch of webhooks! While we looked at 100+ in our own research, we only had time to add ~50 to this initial pass. If you'd like to add your favorites, pull requests welcome: https://github.com/ngrok/webhooks.fyi https://github.com/ngrok/webhooks.fyi And yes, seriously, you can use ngrok to verify webhooks pretty easily: ngrok http 80 --verify-webhook=slack --verify-webhook-secret=[secret]
- tasn 4y ago
  Oh, I didn't know about the new --verify-webhook addition to ngrok, that's cool! Who can I chat with about adding Svix style webhooks[1] to ngrok? We do webhook sending as a service and power the webhooks for many companies, including one of the only three that got a "perfect score" on your list. Adding Svix will make ngrok automatically support many webhooks services. I'd be happy to chat further my email is my name at the domain of the Svix service. [1] https://docs.svix.com/receiving/verifying-payloads/how-manual https://docs.svix.com/receiving/verifying-payloads/how-manua...
  - deleted 4y ago
    [deleted]
dgudkov 4y ago
If only there was a single standard for webhook subscription verification. Cloud services invent their own authentication protocols. Webhook verification in Dropbox is different from that in Trello. The lack of a single standard makes it hard to design a universal incoming webhook service (we faced this problem).
- tasn 4y ago
  Amen. FWIW, we are open source, and we publish libraries that let anyone sign and verify webhooks, hoping to save people the work and have a unified standard. Libs: https://github.com/svix/svix-webhooks https://github.com/svix/svix-webhooks Docs: https://docs.svix.com/receiving/verifying-payloads/how-manual https://docs.svix.com/receiving/verifying-payloads/how-manua... The docs mention `Svix-Signature`, but we (and the libs) also support `Webhooks-Signature` which are useful for the generic standard. The fact that we don't have REST for webhooks is both silly and annoying. Especially since (as you can see on the list in the original post) many companies who implement webhooks end up doing it poorly.
- caseysoftware 4y ago
  I love the idea of a standard but it gets colorful as you start digging in what the individual leaders are doing. On the signature front, there's this but it's not enough: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures-11 https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-mes...
- leetrout 4y ago
  One option is to use a platform like we created at https://hookdeck.com/ https://hookdeck.com/ where we handle verification with multiple providers and then offer downstream destinations a verifiable request (the Hookdeck signature is a SHA-256 based HMAC digest). Drop Hookdeck in as a proxy between the origin service and your destination and you get seamless integrations that help with reliability, observability, and recovery (request replays).
babbledabbler 4y ago
Seeing this right after finishing a webhook integration.
leetrout 4y ago
This is a fantastic resource! Thank you to the folks at ngrok for putting this together! As this site makes clear: webhooks are harder than they appear. Even just consuming webhooks it's easy to get bogged down dealing with issues around rate limits or recovering from bugs that cause missed events! Missed events being particularly painful with platforms that don't offer replay / retry. Disclaimer: I work at https://hookdeck.com/ https://hookdeck.com/ & I shamelessly plug our tool for giving you an awesome developer experience working with webhooks and helping deal with some of the concerns brought up on webhooks.fyi. And if you are interested in webhooks at large a couple more resources worth checking out is the awesome-webhooks[1] list and the r/webhooks[2] subreddit (I just got ownership of the sub and started dusting it off this week after being neglected for the past few years! Please, come join!) [1] https://github.com/realadeel/awesome-webhooks https://github.com/realadeel/awesome-webhooks [2] https://www.reddit.com/r/webhooks/ https://www.reddit.com/r/webhooks/

235 points by m3h 4y ago

Traubenfuchs 4y ago

Why are callback registration and async APIs now called webhooks?

deleted 4y ago
[deleted]
osullip 4y ago
Not sure of the evolution, but maybe because 'callbacks' exist everywhere, but 'webhooks' are specific to REST APIs.
pavlov 4y ago
The usual nomenclature is that regular callbacks and async API calls are made and registered in the same process space as the caller. (E.g. passing a function pointer in C, or using “await” in modern languages.) Webhooks are callbacks received over HTTP from a public API. They may not be programmatically registered at all, e.g. the callback URL may be configured manually in an admin dashboard. So the application has less control over webhooks vs. regular callbacks or async functions.

cuu508 4y ago

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be able to deal with that. * Timeouts (slowloris): the webhook target could be sending back one byte at a time, with 1 second pauses inbetween. If you are using, say, the "requests" python library for making HTTP requests, the "timeout" parameter will not help here * Private IPs and reserved IPs: you probably don't want users defining webhooks to http://127.0.0.1:<some-port> http://127.0.0.1:<some-port> and probing your internal network. Remember about private IPv6 ranges too * Domains that resolve to private IPs: attacker could set up foo.com which resolves to a private IP. It is not enough to just validate webhook URLs when users set them up. * HTTP redirects to private IPs. If your HTTP client library follows HTTP redirects, the attacker can set up a webhook endpoint that redirects to a private IP. Again, it is not enough to validate the user-supplied URL. * Excessive HTTP redirects. The attacker can set up a redirect loop - make sure this does not circumvent your timeout setting. My current solution for all of the above is to use libcurl via pycurl. I wrote a wrapper that mimics requests API: https://github.com/healthchecks/healthchecks/blob/master/hc/lib/curl.py https://github.com/healthchecks/healthchecks/blob/master/hc/... (may contain bugs, use at your own risk :-)

technion 4y ago
Just adding to that - don't forget about users defining AWS metadata addresses for a webhook. Returning IAM data to them can be .. bad.
m3h 4y ago
It seems like webhooks have enough corner cases for the sender to require a specialized tool to protect itself from malicious users and to stay performant. Does anyone have suggestion for such tools/services that they might have used in production?
- calyhre 4y ago
  At Slite, for all outgoing calls we use a sandboxed proxy. It has saved us a few times already. We detailed the trick in a blog post -> https://slite-tech-blog.ghost.io/anti-ssrf-solution/ https://slite-tech-blog.ghost.io/anti-ssrf-solution/
  - freeqaz 4y ago
    This is the path I've seen be fairly robust at a few tech companies I've helped sort out this defense for. I've helped write libraries too but the proxy is the easiest approach when targeting many languages.
- charcircuit 4y ago
  For half of those it's called a firewall and for the other half you should specify timeouts / max redirects.
- rajamaka 4y ago
  Outgoing webhooks dispatched from a Lambda seemed to solve most of the above problems for us
  - tasn 4y ago
    Using lambda doesn't protect you against any of these. Timeout attacks: instead of failing you'll now be just be paying through the nose. And also with enough scale you can hit lambda limits and fail. SSRF: you are still as vulnerable unless the lambda is outside your VPC (but then it's the VPC that solved it, not the lambda).

sparsely 4y ago

As this page makes very clear, it's actually pretty hard to make a robust webhooks implementation! What alternatives are there? I've looked at: * Publishing AWS EventBridge events to other accounts. * /events instead of webhooks https://blog.sequin.io/events-not-webhooks/ https://blog.sequin.io/events-not-webhooks/ * ???

floucky 4y ago
Emails maybe ? But it means you have to regularly check your inbox. Or use a webhook called by your email provider when you receive an email :D
tasn 4y ago
We built Svix[1] exactly for this reason. We make it super easy for companies to send webhooks reliably. There are only three companies with a "perfect score" on webhooks.fyi's list, we power one of them. ;) [1] https://www.svix.com https://www.svix.com
- caseysoftware 4y ago
  That's awesome and glad to see others making it easier and better to do this right. Obviously you should add some of your other customers to make sure they get the attention they deserve. Also, there are other patterns or practices that should be in the list, let us know. PRs are welcome but feel free to email me directly: danger@ngrok.com
  - tasn 4y ago
    Will email you now in a bit! I also just replied to your other comment about another topic, so there's much to discuss. :)
Traubenfuchs 4y ago
> What alternatives are there? At the top of my head: Short Polling, long polling, websockets, response streaming, SSE, SNS, paging API, comet...

egorfine 4y ago

Excellent. I'm in process of building a service delivering notifications via webhooks right now. Thank you!

tasn 4y ago
Shameless plug: we make it super easy for companies to send webhooks. https://www.svix.com https://www.svix.com There are only 3 companies on webhooks.fyi's list with 3 green checks, and we're powering webhooks for one of those. :)
- egorfine 4y ago
  Thank you! Looks promising!

danpalmer 4y ago

Great resource! I’ve encountered most of these problems at some point. It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.

tasn 4y ago
I recorded a video with advice about designing a webhooks system a couple of months ago: https://www.svix.com/blog/webhook-architecture-design/ https://www.svix.com/blog/webhook-architecture-design/ It's high level, so it doesn't touch everything you mentioned, but it's a good primer.

caseysoftware 4y ago

Hello all! I'm one of the creators of webhooks.fyi over at https://ngrok.com/ https://ngrok.com/ Happy to answer any questions! First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole. Next, yes it's ridiculously hard to build webhooks correctly. There are so many shortcuts that feel "okay" and you don't really think about until you realize that it didn't quite work as you had planned and now you have a gap. We're hoping the site can help people move more of that thinking earlier and make better decisions. Finally, we're missing a bunch of webhooks! While we looked at 100+ in our own research, we only had time to add ~50 to this initial pass. If you'd like to add your favorites, pull requests welcome: https://github.com/ngrok/webhooks.fyi https://github.com/ngrok/webhooks.fyi And yes, seriously, you can use ngrok to verify webhooks pretty easily: ngrok http 80 --verify-webhook=slack --verify-webhook-secret=[secret]

tasn 4y ago
Oh, I didn't know about the new --verify-webhook addition to ngrok, that's cool! Who can I chat with about adding Svix style webhooks[1] to ngrok? We do webhook sending as a service and power the webhooks for many companies, including one of the only three that got a "perfect score" on your list. Adding Svix will make ngrok automatically support many webhooks services. I'd be happy to chat further my email is my name at the domain of the Svix service. [1] https://docs.svix.com/receiving/verifying-payloads/how-manual https://docs.svix.com/receiving/verifying-payloads/how-manua...
- deleted 4y ago
  [deleted]

dgudkov 4y ago

If only there was a single standard for webhook subscription verification. Cloud services invent their own authentication protocols. Webhook verification in Dropbox is different from that in Trello. The lack of a single standard makes it hard to design a universal incoming webhook service (we faced this problem).

tasn 4y ago
Amen. FWIW, we are open source, and we publish libraries that let anyone sign and verify webhooks, hoping to save people the work and have a unified standard. Libs: https://github.com/svix/svix-webhooks https://github.com/svix/svix-webhooks Docs: https://docs.svix.com/receiving/verifying-payloads/how-manual https://docs.svix.com/receiving/verifying-payloads/how-manua... The docs mention `Svix-Signature`, but we (and the libs) also support `Webhooks-Signature` which are useful for the generic standard. The fact that we don't have REST for webhooks is both silly and annoying. Especially since (as you can see on the list in the original post) many companies who implement webhooks end up doing it poorly.
caseysoftware 4y ago
I love the idea of a standard but it gets colorful as you start digging in what the individual leaders are doing. On the signature front, there's this but it's not enough: https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-message-signatures-11 https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-mes...
leetrout 4y ago
One option is to use a platform like we created at https://hookdeck.com/ https://hookdeck.com/ where we handle verification with multiple providers and then offer downstream destinations a verifiable request (the Hookdeck signature is a SHA-256 based HMAC digest). Drop Hookdeck in as a proxy between the origin service and your destination and you get seamless integrations that help with reliability, observability, and recovery (request replays).

babbledabbler 4y ago

Seeing this right after finishing a webhook integration.

leetrout 4y ago

This is a fantastic resource! Thank you to the folks at ngrok for putting this together! As this site makes clear: webhooks are harder than they appear. Even just consuming webhooks it's easy to get bogged down dealing with issues around rate limits or recovering from bugs that cause missed events! Missed events being particularly painful with platforms that don't offer replay / retry. Disclaimer: I work at https://hookdeck.com/ https://hookdeck.com/ & I shamelessly plug our tool for giving you an awesome developer experience working with webhooks and helping deal with some of the concerns brought up on webhooks.fyi. And if you are interested in webhooks at large a couple more resources worth checking out is the awesome-webhooks[1] list and the r/webhooks[2] subreddit (I just got ownership of the sub and started dusting it off this week after being neglected for the past few years! Please, come join!) [1] https://github.com/realadeel/awesome-webhooks https://github.com/realadeel/awesome-webhooks [2] https://www.reddit.com/r/webhooks/ https://www.reddit.com/r/webhooks/