6 ms·
Launch HN: Convoy (YC W22) – Open-source cloud-native webhooks service
Hi HN, we’re Subomi and Emmanuel, founders of Convoy (https://getconvoy.io https://getconvoy.io). Convoy is an open-source webhooks service. We make it easy for developers to publish webhooks like Stripe and Twilio. Convoy is cloud-native so developers can deploy the container to wherever they host their applications.
Some time ago I built an API for local fintech companies alongside Emmanuel. While trying to sell this product, we were asked several times for webhooks. We didn't find a great tool that was easy to use, language-agnostic and cloud-native; something that could scale independently of our core service. We decided to build it ourselves.
Webhooks are the glue of the modern internet. They are used for asynchronous communication by API providers to notify apps/integrators about events that have occurred in their account. I like to say it as: if you make a payment on Shopify with Stripe’s checkout and Stripe fails to send a webhook event to Shopify’s servers about your payment, you won’t get a delivery of your package even though your card has been debited. This explains the importance of webhook events to many consumer apps.
Today, implementing a webhooks delivery system is a highly fragmented problem. Stripe's system is designed for security (with features like replay attacks, rolling secrets etc), Twilio for performance (with features like connection override), Pagerduty for flexibility (with features like payload filtering) and many more. We'd love to make these features available to everyone with Convoy. We're already helping a few companies solve problems they didn't know they had because of the visibility Convoy gives.
API providers need to push webhook events reliably. It’s a pain to build this from scratch. At first glance, it seems like good old HTTP POST, but with time it becomes more tricky as the requirements pile up: dealing with bad endpoints, pushing events to multiple endpoints as configured by your user, rate-limiting of customer endpoints, security of delivery, UX for managing retries. We take care of all that and give you one less infrastructure to worry about.
Convoy is open-source software that exposes a REST API. Developers push events to it using the REST API (we aim to support other means of ingesting data in the future for high volume environments – this is a feature Shopify provides today) then we sign the payload, publish the events, apply retries, etc. We also provide a management UI to manually retry events to user endpoints.
Convoy consists of 3 core components—in addition to the REST API, there is a job queue and a storage layer. The REST API server is used to create applications, endpoints and events. The job queue currently supports 2 backends: in-memory and Redis. The storage layer currently supports 2 backends: on-disk and MongoDB. All events pushed to Convoy are saved to storage and enqueued on the job queue for workers to deliver.
Most of our users use our product to push webhook events to their customers’ endpoints. Customers also use Convoy as a broker for inter-service communication.
We make money by providing a managed service. We've been running the managed service for a while now and charge $1/5k events, but those details are not on our website yet.
The repository is available at https://github.com/frain-dev/convoy https://github.com/frain-dev/convoy and a getting started doc exists here: https://getconvoy.io/docs/guide https://getconvoy.io/docs/guide. Anyone can download the binary or Docker image from here: https://getconvoy.io/download https://getconvoy.io/download and run it wherever they choose - AWS, Azure, GCP, etc.
We’d love you to try it out and hear your thoughts! We’d also love to hear about your experiences building custom webhook software and the challenges you’ve faced?
- toomuchtodo 5y agoCongrats on the launch, huge space with lots of product market fit to go around, value prop is very clear. Do you have any plans to support PostgreSQL or an S3 compatible target as storage backends?
- subomi 5y agoThank you! Yes, we have plans to support PostgreSQL as a storage backend. At the moment, queue tuning parameters are not exposed as configuration parameters, they're based off a shared queue and we expose prometheus metrics over the shared queue. What use-cases do you have for tuning the queue?
- toomuchtodo 5y agoI might not have enough context, but is it possible based on your architecture the shared queue could back up if specific destination endpoints aren't accessible when workers are processing the queue (I have seen this happen before in an adjacent space product)? If this isn't the case, tuning is probably not necessary if temporarily undeliverable messages can be skipped and retried later without impacting queue throughput. Being able to select, group and pause processing of queued messages in an ad hoc fashion (tuning was likely the wrong word, my apologies), I've found, is a powerful feature of webhook engines.
- subomi 5y agoOh, yes. So essentially Convoy's queue mechanism works as a delayed job queue (basically how retries in webhooks work e.g. Stripe will send the event to you every hour for the next 72 hours). When the endpoint is unaccessible it backs up and schedules the retries at a future time on the queue. The implementation under the hood is a combination of redis zset + redis streams. :) When it maxs out retries the events are still stored in the DB and you can batch retry them once the endpoints become accessible.
- toomuchtodo 5y ago