4 ms·
Looks like this could be useful for webhooks delivery, a problem our team is working on at the moment. We have 100's of tenants, each of which could get their
by clon 8y ago
Looks like this could be useful for webhooks delivery, a problem our team is working on at the moment.
We have 100's of tenants, each of which could get their own stream with some delivery guarantees set. If one tenant's endpoint is down or another tenant fills the stream with 1M messages, it should not affect delivery rates of other tenants. Seems to fit the bill.
I see the HTTP output mentions some retries, but I guess these run as part of the delivery step, blocking this goroutine as opposed to rescheduling the message? Sometimes it takes hours for clients to restore their receiving systems and it would be great if messages for past N hours would still be delivered..
- andy_ppp 8y agoI'd strongly suggest using Elixir for this but then I'm a bit of a fan boy ;-) https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections https://phoenixframework.org/blog/the-road-to-2-million-webs...
- jeffail 8y agoHey, the HTTP output has a fixed number of retries, after which you could either have some mechanism in place to fall back on or by default it will simply continue the retries again whilst blocking upstream. You might also be interested in running Benthos in streams mode: https://github.com/Jeffail/benthos/tree/master/docs/streams https://github.com/Jeffail/benthos/tree/master/docs/streams Streams mode lets you run as many isolated stream pipelines as you want in the same process, which in your case could be a simple queue -> webhook bridge. You can manage these pipelines either statically in config files, or dynamically through a REST API.
- clon 8y agoThat seems really promising, especially configuring pipelines with API calls. Many wheels may be left uninvented. Thank you for making your work available!