3 ms·
I'm also curious how this compares to Celery with rate limiting. One advantage of Celery is that you can use an AMQP broker which is probably better suited than
by armon 12y ago
I'm also curious how this compares to Celery with rate limiting. One advantage of Celery is that you can use an AMQP broker which is probably better suited than Redis if you care about durability and at-least-once delivery.
- ohmygeek 12y agoIIRC, Celery uses token bucket algorithm for rate limiting. SHARQ uses the leaky bucket algorithm (http://en.wikipedia.org/wiki/Leaky_bucket#The_Leaky_Bucket_Algorithm_as_a_Queue http://en.wikipedia.org/wiki/Leaky_bucket#The_Leaky_Bucket_A...). The advantage of leaky bucket is that, it enforces a constant flow of jobs out of SHARQ regardless of the spike in enqueue rate. Even though SHARQ uses Redis, it has been designed in a way to support at-least-once delivery. If you see the getting started section (http://sharq.io/docs/gettingstarted.html http://sharq.io/docs/gettingstarted.html), there is a Finish API which would acknowledge every job dequeued. SHARQ ensures that a job will be requeued back into the queue if it does not receive a Finish request.