2 ms·
Even with separate queues per customer, how are you ensuring ordering? Are you using something like Kafka which ensures ordering with-in partitions?
by tirumaraiselvan 8y ago
Even with separate queues per customer, how are you ensuring ordering? Are you using something like Kafka which ensures ordering with-in partitions?
- Diggsey 8y agoNo, Kafka is one of the few messages queues which supports ordering, so for a more advanced solution it could be the right choice, but we don't use anything more than postgres. We have workers "take locks" on the queues they are processing. I put that in quotes because they are purely software locks - workers store the time when they started working on the queue, and other workers avoid queues which have a time less than X seconds ago. When a worker finishes with a queue it resets the timestamp to NULL. It's still possible for a worker to take too long to send a webhook and for another worker to pick it up (although very unlikely because we can set X quite large with no side effects): in that case the worst that can happen is that the webhook is sent twice, so we can still guarantee that the first successful send of each webhook occurs in the right order. Our webhooks all have UUIDs, so it's pretty easy for consumers to process them idempotently.