7 ms·
Webhooks do’s and dont’s: what we learned after integrating APIs
- nimblegorilla 10y agoIt's also really handy when API providers give a nice webhook UI that lets you view and resend webhooks during development.
- giuliano84 10y agoTrue. Also having a sample request at the very beginning is useful so you don't have to find a way to trigger the event by clicking around on the product UI.
- onion2k 10y agoBitBucket is a very good example of web hook integrations done right. Relatable, logged, and well documented. I learned from their UI when I implemented my own version.
- deleted 10y ago[deleted]
- misterbowfinger 10y agoThis is going to sound bizarre, but why do webhooks and not just an AMQP queue? I get that receiving HTTP POSTs is easier, but it just seems better to setup a publisher/subscriber relationship. That way, if a subscriber goes down, they can always catch up. And publishers can allow messages to sit in the queue with a TTL and max_size. It seems like a win-win for everyone.
- elmigranto 10y agoThough this is an approach worth considering for a bunch of local services, at least a subset of them, I don't see it working with 3rd party APIs. Consider something like Stripe — it is probably far more straightforward to invoke some HTTP endpoints, than to set up a huge infrastructure with millions persistent client connections.
- giuliano84 10y agoAgreed. I believe that AMQP connections are best suited for wide and articulated local environments. Moreover REST HTTP endpoints are today's esperanto :)
- jbert 10y agoOne difference is in "dead connection detection". How do you know that your AMQP connection is down? At some level you're polling, whether that be TCP keepalive, application keepalive or something else. If you're doing polling, you're actually back at the same pre-webhook place - polling their server on some timescale which is a compromise between latency and load. Yes, a TCP keepalive is generally cheaper than an HTTP long poll request, but only by a constant factor.
- jon-wood 10y agoIt's not AMQP (sadly) but something I've done previously is to have the actual webhook endpoint be as dumb as possible, doing nothing but accepting the payload (maybe with some very high level validation that the request was expected) and pushing it into a real queueing system. This means you can handle all sorts of failure modes, not just the backend going down, but also bugs in the consumer that would otherwise result in losing the request. I've not tried it, but I imagine this is a pretty good usecase for AWS Lambda as it's a small bit of glue code.
- fiatjaf 10y agoShameless plug: https://requesthub.xyz https://requesthub.xyz is ideal for these cases.
- jon-wood 10y agoThat's pretty awesome, thanks for the tip.
- stephenr 10y agoI've used the same basic concept for accepting payment-received notifications (from a http redirect based payment gateway): Read the transaction ID from the request body, and store it (with a date/time) in a table. A periodic process later checks them, and uses the payment service's API to validate the payment is valid and take appropriate action.
- niftich 10y agoBecause anyone can throw another route onto port 443 if you already host a website. A non-HTTP protocol running on a dedicated port, despite often being a superior solution, requires extra effort to set up, if the hosting environment even provides that option.
- morgante 10y agoIt's a whole lot simpler to do secure cross-organization HTTP requests than it is to figure out how to have multiple AMQP subscribers from untrusted companies.
- shizcakes 10y agoI think the "securing webhooks" section is missing some critical tips that we've learned in production. 1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses. 2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the person configuring the webhook. Say there was an error, what the code was, etc, but returning the response body means you basically just gave someone curl with a starting point on your network (see 1 as well) 3) Find ways to perform other validations of those webhooks. Are the URLs garbage? Are they against someone else's system? Create validation workflows that require initial pushes to the URL with a validation token to be entered back into your system, like validating an email address by clicking a link.
- detaro 10y agoThe entire guide seems mostly consumer-centric: What do they as the one being called want, sender-side concerns are missing entirely.
- giuliano84 10y agoYes but that's why you build webhook for, to let people consume your content. I think DX is critical today and big companies can afford to do the heavy-lifting. As I mentioned in the Subscription Expiration paragraph I totally get Microsoft's reason to put a 72hrs expire date on subscriptions but it adds some friction on the consumer side.
- deleted 10y ago[deleted]
- qyv 10y agoConsumer convenience cannot be made at the expense of security measures and abuse mitigation, see: IoT.
- giuliano84 10y agoPoint 3 is spot on. I think it would be a good strategy to avoid expire dates on subscriptions. Producers could take decisions on whether or not keep sending data by monitoring responses on the consumer's target URL. Another thing that it should be worth mentioning is that some services batch notifications (e.g Facebook Messenger) so that they can send more data in a single POST request.
- deleted 10y ago[deleted]
- madamelic 10y agoAlso, in your documentation, please show what the webhook events will look like since developers actually want to write code and not guess at what we will get. cough Stripe. (https://stripe.com/docs/api#events https://stripe.com/docs/api#events)
- enraged_camel 10y agoYou don't have to guess anything. Stripe lets you send test webhooks to the endpoint you specify[1]. You can set up something like ngrok[2] on your localhost to examine the headers and bodies, then write your code to parse them accordingly. [1]http://i.imgur.com/oCpxYwE.png http://i.imgur.com/oCpxYwE.png [2]https://ngrok.com/ https://ngrok.com/
- madamelic 10y agoYeah, that is one method. I also learned about services that will set up test webhooks without having to go about setting up a server, etc. I think I might use that, but I still think that docs should at least explain what will get sent. Maybe that is a bit too verbose in Stripe's case though.
- brandur 10y agoThe implication was meant to be that the information under `data/object` is simply a full representation of another API resource of the type on which the event occurred, and that you can look elsewhere in the documentation to see exactly what each type will look like (you can see a subscription embedded in the sample response for example). Fair enough that we could rewrite this to be more explicit about that though! We'll see what we can do to make that section more clear. (I work for Stripe.)
- madamelic 10y agoThis is what I love. I purposefully talk about Stripe on here just because I know Stripe people browse HN. Stripe is great though. :)
- arxpoetica 10y agoHonest question. Why webhooks over something like push/pull handshaked socket?
- detaro 10y agoFor the receiver: Everything that can run a dynamic website can run a webhook receiver, opening an arbitrary socket connection isn't possible in all environments (e.g. software running on shared hosting or PaaS). You'd also need to define and implement a protocol on top of said socket, whereas more or less every web developer knows what to do with HTTP POST with a JSON payload. And for the sender, keeping many concurrent connections open can be quite a challenge. Sending Webhooks also takes resources, but at least you can easily distribute it over many machines/processes if necessary.
- ec109685 10y agoSlack offers both. Their realtime API is especially nice because you connect to it rather than needing to deploy public facing web services to receive events from them. On the other hand, if you are not connected, messages could be lost unless you build in syncing capability, whereas with web hooks, Slack will handle the retries for you.
- bazizbaziz 10y agoHow do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to address this issue is to use the webhook as a hint that you need to run some other process that guarantees you've got all updates.
- madamelic 10y agoStripe has a retry policy as well. You can set up something where it will alert you if there are too many failures in a certain time period. That isn't offered by Stripe but you can build it. If you mean in the case of "catastrophic failure", there is none. If there is a "catastrophic failure" (machine gets shut off for a week, data center blown up, whatever), there are probably bigger issues or we probably would already know.
- brandur 10y agoStripe has an "events" API that can be polled to receive the same content that you would have received via Webhook [1]. (Disclaimer: I work there.) If you missed some Webhooks due to an application failure, it's possible to page through it and look for omissions. I've spoken to at least one person integrating who had this sort of setup running as a regular process to protect against the possibility of dropped Webhooks. This usually works pretty well, but does start to break down at very large scale where events are being created faster than you can page back. The possibility of dropped events is a major disavantage of Webhooks in my mind -- if you consider other alternatives for streaming APIs like a Kafka/Kinesis-like stream (over HTTP) that's simply iterated through periodically with a cursor, you avoid this sort of degenerate case completely, and also get nice things like a vastly reduced number of total HTTP requests, and guaranteed event ordering. (But to be clear, Webhooks are overall pretty good.) [1] https://stripe.com/docs/api#events https://stripe.com/docs/api#events
- 10y ago
- sly010 10y agoAside: Webhooks are always a pain. Implementing polling is easier for both sides. I routinely have to integrate with random 3rd party systems, some with no or broken webhooks, some with no API at all. It turns out for my customers (this may not be always the case) eventual consistency is more important than timelyness. What I do now every time I need to sync data from a third party is I always implement some sort of pull first with idempotent logic on my side. It's easier, and it allows me to just re-run things if something fails (e.g. network error, unexpected data in production, etc). Only when that works reliably and only if required by the customer I implement a webhook, but I usually throw away most of the message and just wake up my polling worker that is otherwise polling relatively slowly.
- abraae 10y agoLong polling works brilliantly (where your API call blocks until there are some results or until timeout occurs - then you loop and call again). Long polling gives you the best of both worlds - easy programming model with instant alerting rather than the delay of normal polling. The only downside really is the need for a more or less permanently open connection per client. As long as the server does not use a naive "thread per connection" model this can scale up to many hundreds of thousands of clients or more.
- Animats 10y agoThe good thing about long polling is that if the connection breaks, the keep-alive will time out and you'll know you're not getting updates. Assuming there's some keep-alive feature.
- MrBuddyCasino 10y agoThats what I did, too. Poll, fetch and remember and retry errors, and if possible implement a sliding window as a poor man's cursor using dateFrom / dateTo if available.
- johns 10y agoI did a talk a little while back on providing a good developer experience around webhooks that covers a lot of the same topics. I wish I had a recording of it, but the slides are here: https://speakerdeck.com/johnsheehan/crafting-a-great-webhooks-experience-1 https://speakerdeck.com/johnsheehan/crafting-a-great-webhook... Edit: found the video https://www.youtube.com/watch?v=xc5ezyJjz1k&feature=youtu.be&t=1266 https://www.youtube.com/watch?v=xc5ezyJjz1k&feature=youtu.be...
- boubiyeah 10y agoWebhook only makes sense if you don't care a single bit about missing updates. If not, it's deeply flawed. A pull model (polling, long-polling, SSE, etc) is strictly superior for synchronisation. You just can't "miss" updates, can restart from the beginning again and reinterpret past events in a different light, the client goes at its own pace, etc.
- icebraining 10y agoLuckily they aren't incompatible, you can take advantage of both.
- paulddraper 10y agoTo expand on icebraining's comment, you can use weebhooks as notifications to poll.
- jarcoal 10y agoSort of disagree with the send-everything-in-the-payload approach. It opens your system up to all sorts of weird edge case bugs like receiving hooks out of order which could mean stale data is considered fresh. It also means you have to care a lot more about verifying the authenticity of the request.
- nimblegorilla 10y ago"trust but verify" Sometimes you want to ignore webhooks based on the payload (or put them into a different queue). It's faster to do that if you get the payload up front.
- abraae 10y agoAgree. Its better to use webhooks as a pure signal that something has changed, and then in the case of update or insert, have the client pull whatever they want using normal API. Otherwise, you end up in a descending vortex of madness trying to specify some protocol whereby the client can specify in advance which properties they care about.
- draaglom 10y agoThis is correct, and important. Webhook payloads need to be logically monotonic[0]; this probably means either: - having a lamport-clock timestamp for each payload so you can entirely discard older payloads in favour of new ones - a well defined / consistent "merge" function over the subset of the payload you care about (e.g. maybe you know a customer's state can never go back from "registered" to "guest") [0]: http://bloom-lang.net/calm/ http://bloom-lang.net/calm/
- Animats 10y agoSo "push technology" is called "webhooks" now? How does this all integrate with HTTP 2? Can you get your notifications over a channel you already have open for other reasons?
- supernintendo 10y agoNot quite. "Push technology" is sort of an all-encompassing term for server-to-client updates whereas webhooks pertain specifically to HTTP callbacks. An example of a webhook would be GitHub making a POST request to some URL (set by the user) whenever new commits are made to a repo. Push technology might take the form of webhooks, long polling, WebSockets etc. Webhooks are traditional HTTP requests so I don't believe HTTP/2 changes anything. The ability to differentiate notifications depends on the service / API you're integrating with.
- g105b 10y agoApostroph'ed.
- mosselman 10y agoIt would have been interesting to read about tests for web-hooks. How do you do integration testing for example?
- stevekemp 10y agoIn the past when I used to support webhooks what I did was very simple: * Receive the HTTP POST submission to my hook end-point. * Save this data in a queue. * Return to the hook-caller "200 OK - $ID". This was better than trying to initiate a long-running job as a result of the hook, and meant that I could trigger "fake webhooks" just by adding data to the queue manually. I'm sure there are other approaches, but this is a flexible one that also gave the benefit of being simple. (For the queue I just used Redis.)
- simo9000 10y agoI'd like to follow up on the statement that the OpenAPI tools do not support webhooks. This is slated to change in an upcoming version of the OpenAPI-specification. Check out https://github.com/OAI/OpenAPI-Specification/pull/763 https://github.com/OAI/OpenAPI-Specification/pull/763 to see details. As soon as this is released, it only be a matter of time before Swagger and the rest support webhooks.
- z3t4 10y agoTo be fully client based (serverless) you need a middle-man for web-hooks. Websockets are a better alternative for stand-alone web clients. There are also "push notifications" via web workers but they are vendor dependent.
- intellent 10y agoWhat I am most interested in is how to test/debug webhooks during development. How do I tell webhook providers to send test notifications to my local development instance without tampering with the production setup on both sides?
- jeffnappi 10y agoThis is another thing that Stripe nails. Out of the box it comes with a Test mode and you can easily use this to test your webhook implementation. Another way to handle this is to create and maintain a mocking tool that will generate requests.