11 ms·

Building Webhooks into Your Application: Guidelines and Best Practices (2020)

115 points by grinich 6y ago

tyronehed 6y ago
f
koblas 6y ago
What people totally omit when thinking about webhooks is the parallels to web sockets and client side updates. In the last round of our system updates, we have "attached" the web sockets and webhooks to the same message bus which has simplified the overall design of the system.
- tasn 6y ago
  Yeah, they share a lot of the same challenges as they are both unlike the rest of the web stack (which is just request -> response). There are a lot of similarities, but a lot of different challenges too.
tasn 6y ago
Hehe, just did a show HN[1] for Diahook (webhooks as a service) earlier today. I wonder if that's what triggered this posting. :P Anyhow, I wish more people had better webhooks, it's such an important part of any API, so please read this blog post, understand the challenges, and provide a great webhooks experience! [1] https://news.ycombinator.com/item?id=26399672 https://news.ycombinator.com/item?id=26399672
- bellttyler 6y ago
  Wow Diahook looks great! My company is in the process of updating our webhooks, so we will be sure to check it out.
  - tasn 6y ago
    Thanks! Feel free to reach out tom @ the domain. :)
alexsmolen 6y ago
Kind of wild that there's no mention of SSRF. A quick search shows it's a pretty frequent security issue in Webhooks: https://www.google.com/search?q=ssrf+webhook https://www.google.com/search?q=ssrf+webhook
- michaelbuckbee 6y ago
  This would be a good addition: make sure that your HTTP library POSTing the webhook won't follow redirects and is blocked from accessing internal domains.
bellttyler 6y ago
This was a great read – thanks for sharing! I've always been a huge fan of how Stripe handles their webhooks. They are pretty much doing everything listed in your article.
mrkurt 6y ago
SSRF is a tremendous problem with webhooks. Stripe also solves this: https://fly.io/blog/practical-smokescreen-sanitizing-your-outbound-web-requests/ https://fly.io/blog/practical-smokescreen-sanitizing-your-ou...
- grinich 6y ago
  Agreed, I should add something on this. (Haven't updated the article since posting it many months ago.)
  - alexsmolen 6y ago
    This is a pretty good article about preventing SSRF including DNS rebinding-based attacks in Go https://www.agwa.name/blog/post/preventing_server_side_request_forgery_in_golang https://www.agwa.name/blog/post/preventing_server_side_reque...
- tasn 6y ago
  Yeah, it's one of the thing that people miss when implementing webhooks naively, and I bet it's one of the things people miss when consuming them!
grinich 6y ago
Hey HN - I'm the author of this post. (Founder of WorkOS) Lots of people have written about how to scale event-based systems or how to design distributed infrastructure for queue management. But what's often overlooked is the subtle-yet-important product decisions that go into designing webhooks for a great developer experience. We don't actually offer webhooks-as-a-service at WorkOS, but we provide webhooks to developers and I've built this feature twice now, previously at Nylas and now at WorkOS. Hopefully this article helps the next person who builds them. :) For anyone curious how we do it at WorkOS, here's our webhooks API reference: https://workos.com/docs/reference/webhooks/connection https://workos.com/docs/reference/webhooks/connection
weddpros 6y ago
I can never understand why enforcing (sane and secure) TLS and adding a (shared) secret request header isn't the usual simple answer. As far as I know, TLS prevents all these security issues, except authentication of the webhook caller, hence the shared secret which your webhooks should check. I've verified in the TLS specs that every issue I've seen raised by people is actually covered, yet people still think (and write) that TLS doesn't sign traffic (it does), or TLS doesn't prevent replay attacks (it does), etc. Usually I give them pointers to the specs, and they say "it doesn't hurt to add even more security" but they're really rolling out their own encryption, which you shouldn't do. I'd say enforce TLS and authenticate the webhook caller...
- LunaSea 6y ago
  TLS with certificate pinning could probably even take care of the authentication issue.
- tasn 6y ago
  I'm the developer of Diahook (we do webhooks as a service), and I agree. It's just so annoying that we have to implement signing ourselves instead of just using client TLS certificates. Though the problem is that there are so many places in the stack that interact with your request before it hits user code (load balancer, web server, etc), that getting anything like this to work is wishful thinking...
cdcarter 6y ago
Not just logging the outbound webhooks to STDOUT and hope they get ingested into logstash or something, but actually log them in your SoR/RDBMS so that you can present that data to a debugging developer.
- koolba 6y ago
  And make sure to impose a max size. Getting a 1GB gzip bomb persisted to your DB is never fun.
olivierduval 6y ago
Maybe a stupid question but... why not using an MQTT server instead of webhooks? Clients could be notified in RT, TLS is already backed in the protocol with client auth... the client doesn't need to expose an endpoint, only to connect to a server...
michaelbuckbee 6y ago
Something else Stripe and most other "serious" webhook senders do is only send Webhooks from a predefined set of IP addresses [1]. This isn't the security guarantee that signing, etc. are but it's a substantial help in adding some Security In Depth. 1 - https://stripe.com/docs/ips https://stripe.com/docs/ips
jaren 6y ago
democratize access to webhooks -- love dat energy
GhvstCode 6y ago
This was an interesting read! If you ever decide to update this article & include tools for working with webhooks, Hookdeck(https://hookdeck.io https://hookdeck.io) should be at the top of the list. Hookdeck is a complete webhook infrastructure that makes working with webhooks painless. we pride ourselves in reliability. We provide tooling for easy monitoring and troubleshooting of webhooks
jakejscott 6y ago
AWS EventBridge announced API destinations recently [1]. I can't find anywhere in their docs about SSRF, but given it's a managed service running outside of your network it sounds like a good option. Anyone care to comment if this would be a good solution for webhooks? 1 - https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/ https://aws.amazon.com/blogs/compute/using-api-destinations-...
winrid 6y ago
At FastComments we send an your API token in the webhook request, and if your API doesn't return a 401 with an invalid token the integration won't activate.
- chrismorgan 6y ago
  I can see why you might do that, but it seems mildly hostile to me and would have broken things for me on at least one or two occasions. I know I’ve set up very simple hook arrangements before where I don’t care about authentication, at least in part because the URL is secret and it’s probably just about dumping the hook call contents for now anyway. But at the same time, perhaps doing this would solve more problems than it creates.
  - winrid 6y ago
    It's not hostile. It's ensuring you properly authenticate the request, and that you own the target endpoint. :)

115 points by grinich 6y ago

What people totally omit when thinking about webhooks is the parallels to web sockets and client side updates. In the last round of our system updates, we have "attached" the web sockets and webhooks to the same message bus which has simplified the overall design of the system.

tasn 6y ago
Yeah, they share a lot of the same challenges as they are both unlike the rest of the web stack (which is just request -> response). There are a lot of similarities, but a lot of different challenges too.

tasn 6y ago

Hehe, just did a show HN[1] for Diahook (webhooks as a service) earlier today. I wonder if that's what triggered this posting. :P Anyhow, I wish more people had better webhooks, it's such an important part of any API, so please read this blog post, understand the challenges, and provide a great webhooks experience! [1] https://news.ycombinator.com/item?id=26399672 https://news.ycombinator.com/item?id=26399672

bellttyler 6y ago
Wow Diahook looks great! My company is in the process of updating our webhooks, so we will be sure to check it out.
- tasn 6y ago
  Thanks! Feel free to reach out tom @ the domain. :)

alexsmolen 6y ago

Kind of wild that there's no mention of SSRF. A quick search shows it's a pretty frequent security issue in Webhooks: https://www.google.com/search?q=ssrf+webhook https://www.google.com/search?q=ssrf+webhook

michaelbuckbee 6y ago
This would be a good addition: make sure that your HTTP library POSTing the webhook won't follow redirects and is blocked from accessing internal domains.

bellttyler 6y ago

This was a great read – thanks for sharing! I've always been a huge fan of how Stripe handles their webhooks. They are pretty much doing everything listed in your article.

mrkurt 6y ago

SSRF is a tremendous problem with webhooks. Stripe also solves this: https://fly.io/blog/practical-smokescreen-sanitizing-your-outbound-web-requests/ https://fly.io/blog/practical-smokescreen-sanitizing-your-ou...

grinich 6y ago
Agreed, I should add something on this. (Haven't updated the article since posting it many months ago.)
- alexsmolen 6y ago
  This is a pretty good article about preventing SSRF including DNS rebinding-based attacks in Go https://www.agwa.name/blog/post/preventing_server_side_request_forgery_in_golang https://www.agwa.name/blog/post/preventing_server_side_reque...
tasn 6y ago
Yeah, it's one of the thing that people miss when implementing webhooks naively, and I bet it's one of the things people miss when consuming them!

grinich 6y ago

Hey HN - I'm the author of this post. (Founder of WorkOS) Lots of people have written about how to scale event-based systems or how to design distributed infrastructure for queue management. But what's often overlooked is the subtle-yet-important product decisions that go into designing webhooks for a great developer experience. We don't actually offer webhooks-as-a-service at WorkOS, but we provide webhooks to developers and I've built this feature twice now, previously at Nylas and now at WorkOS. Hopefully this article helps the next person who builds them. :) For anyone curious how we do it at WorkOS, here's our webhooks API reference: https://workos.com/docs/reference/webhooks/connection https://workos.com/docs/reference/webhooks/connection

weddpros 6y ago

I can never understand why enforcing (sane and secure) TLS and adding a (shared) secret request header isn't the usual simple answer. As far as I know, TLS prevents all these security issues, except authentication of the webhook caller, hence the shared secret which your webhooks should check. I've verified in the TLS specs that every issue I've seen raised by people is actually covered, yet people still think (and write) that TLS doesn't sign traffic (it does), or TLS doesn't prevent replay attacks (it does), etc. Usually I give them pointers to the specs, and they say "it doesn't hurt to add even more security" but they're really rolling out their own encryption, which you shouldn't do. I'd say enforce TLS and authenticate the webhook caller...

LunaSea 6y ago
TLS with certificate pinning could probably even take care of the authentication issue.
tasn 6y ago
I'm the developer of Diahook (we do webhooks as a service), and I agree. It's just so annoying that we have to implement signing ourselves instead of just using client TLS certificates. Though the problem is that there are so many places in the stack that interact with your request before it hits user code (load balancer, web server, etc), that getting anything like this to work is wishful thinking...

cdcarter 6y ago

Not just logging the outbound webhooks to STDOUT and hope they get ingested into logstash or something, but actually log them in your SoR/RDBMS so that you can present that data to a debugging developer.

koolba 6y ago
And make sure to impose a max size. Getting a 1GB gzip bomb persisted to your DB is never fun.

olivierduval 6y ago

Maybe a stupid question but... why not using an MQTT server instead of webhooks? Clients could be notified in RT, TLS is already backed in the protocol with client auth... the client doesn't need to expose an endpoint, only to connect to a server...

michaelbuckbee 6y ago

Something else Stripe and most other "serious" webhook senders do is only send Webhooks from a predefined set of IP addresses [1]. This isn't the security guarantee that signing, etc. are but it's a substantial help in adding some Security In Depth. 1 - https://stripe.com/docs/ips https://stripe.com/docs/ips

jaren 6y ago

democratize access to webhooks -- love dat energy

GhvstCode 6y ago

This was an interesting read! If you ever decide to update this article & include tools for working with webhooks, Hookdeck(https://hookdeck.io https://hookdeck.io) should be at the top of the list. Hookdeck is a complete webhook infrastructure that makes working with webhooks painless. we pride ourselves in reliability. We provide tooling for easy monitoring and troubleshooting of webhooks

jakejscott 6y ago

AWS EventBridge announced API destinations recently [1]. I can't find anywhere in their docs about SSRF, but given it's a managed service running outside of your network it sounds like a good option. Anyone care to comment if this would be a good solution for webhooks? 1 - https://aws.amazon.com/blogs/compute/using-api-destinations-with-amazon-eventbridge/ https://aws.amazon.com/blogs/compute/using-api-destinations-...

winrid 6y ago

At FastComments we send an your API token in the webhook request, and if your API doesn't return a 401 with an invalid token the integration won't activate.

chrismorgan 6y ago
I can see why you might do that, but it seems mildly hostile to me and would have broken things for me on at least one or two occasions. I know I’ve set up very simple hook arrangements before where I don’t care about authentication, at least in part because the URL is secret and it’s probably just about dumping the hook call contents for now anyway. But at the same time, perhaps doing this would solve more problems than it creates.
- winrid 6y ago
  It's not hostile. It's ensuring you properly authenticate the request, and that you own the target endpoint. :)