3 ms·

I might be missing something but it seems like all of your details are either things you would need to configure anyways in Svix (not all services should have t

I might be missing something but it seems like all of your details are either things you would need to configure anyways in Svix (not all services should have the same retry/expiry) or things that are not solved by this service. This service takes HTTP as input and output, so you wouldn't need a worker per topic anyway, right? The workload is http-in, http-out, with a failure condition for retry. If I already have a queue of http messages (which I need to have to protect from Svix downtime) configured with their policies for retry/expiry (which I need to configure since it's not the same for all) then what does this service do that is not basically a curl loop with an error check?

tasn 5y ago
You don't need a queue to protect from Svix downtime. It can be as simple as logging failures to svix (when they happen), and replaying these events. Though as I said elsewhere, this scenario is something you'd need to deal with Twilio and SendGrid too. As for what this service does that is not basically a curl loop with an error check: see the rest of the comments. People chimed him from their experience better than I could have said it myself. Or even look at https://svix.com https://svix.com and see what we offer, you'll see that there's much more nuance. :) We know that people underestimate webhooks, it's a challenge we need to overcome, but there really is more to it than just a POST request.
- SahAssar 5y ago
  > It can be as simple as logging failures to svix (when they happen), and replaying these events That's a manually implemented queue, right? I looked at the site and this thread and I still don't get it, I don't think I underestimate webhooks, but rather that I don't see why adding another webhook inbetween will help.
  - tasn 5y ago
    It's more of an append-only failure log than a queue, which is a whole different beast... Though as I said elsewhere in the thread, the actual delivery is just part of what we do.
atombender 5y ago
But a queue to protect against Svix downtime is fundamentally different from delivering webhooks. I already outlined some challenges with implementing webhooks. I think you're missing my point about parallel delivery. If the workload is "HTTP-in, HTTP-out", you need to make sure that a single slow "out" does not cause head-of-line blocking that would prevent other, fast workloads from being executed. One way to accomplish that is to scale up to have N_workers >= N_pending, which is typically a terrible solution. So a mature webhook solution needs to be more clever about this. Queues are great for situations where either the latency doesn't matter, or where you can scale up your resources to decrease latency; but in the case of webhooks, the latency of the webhook receiver is outside your control — you can't scale them up. Here's another detail where devils are hiding: Delivering webhooks to arbitary URLs is a security concern. The mitigate this, the delivery agent run in an isolated environment so that it cannot possibly interfere with private hostnames/IPs in your cluster.

tasn 5y ago

You don't need a queue to protect from Svix downtime. It can be as simple as logging failures to svix (when they happen), and replaying these events. Though as I said elsewhere, this scenario is something you'd need to deal with Twilio and SendGrid too. As for what this service does that is not basically a curl loop with an error check: see the rest of the comments. People chimed him from their experience better than I could have said it myself. Or even look at https://svix.com https://svix.com and see what we offer, you'll see that there's much more nuance. :) We know that people underestimate webhooks, it's a challenge we need to overcome, but there really is more to it than just a POST request.

SahAssar 5y ago
> It can be as simple as logging failures to svix (when they happen), and replaying these events That's a manually implemented queue, right? I looked at the site and this thread and I still don't get it, I don't think I underestimate webhooks, but rather that I don't see why adding another webhook inbetween will help.
- tasn 5y ago
  It's more of an append-only failure log than a queue, which is a whole different beast... Though as I said elsewhere in the thread, the actual delivery is just part of what we do.

atombender 5y ago

But a queue to protect against Svix downtime is fundamentally different from delivering webhooks. I already outlined some challenges with implementing webhooks. I think you're missing my point about parallel delivery. If the workload is "HTTP-in, HTTP-out", you need to make sure that a single slow "out" does not cause head-of-line blocking that would prevent other, fast workloads from being executed. One way to accomplish that is to scale up to have N_workers >= N_pending, which is typically a terrible solution. So a mature webhook solution needs to be more clever about this. Queues are great for situations where either the latency doesn't matter, or where you can scale up your resources to decrease latency; but in the case of webhooks, the latency of the webhook receiver is outside your control — you can't scale them up. Here's another detail where devils are hiding: Delivering webhooks to arbitary URLs is a security concern. The mitigate this, the delivery agent run in an isolated environment so that it cannot possibly interfere with private hostnames/IPs in your cluster.