4 ms·

I'm the primary author SHARQ. Appreciate any feedback.. Here is the link to the slides from my PyCon India talk: https://speakerdeck.com/sandeepraju/sharq-an-o

I'm the primary author SHARQ. Appreciate any feedback.. Here is the link to the slides from my PyCon India talk: https://speakerdeck.com/sandeepraju/sharq-an-open-source-rate-limited-queueing-system-pycon-india-2014 https://speakerdeck.com/sandeepraju/sharq-an-open-source-rat... Here is our blog announcement: https://www.plivo.com/blog/sharq-a-flexible-open-source-rate-limited-queuing-system/ https://www.plivo.com/blog/sharq-a-flexible-open-source-rate...

wc- 12y ago
Looks pretty cool, looking forward to playing around with it. How does this compare to celery w/ rate limiting of individual tasks? Celery can be pretty daunting (heavyweight perhaps?) sometimes, having a simpler alternative could be useful. However, I wonder if by the time you need rate limiting of queues, you will probably need additional "heavy" features as well. I'd love to see a deeper comparison of sharq's usage of redis vs celery's. For example, are there any performance gains by using sharq's lua commands or schemas vs celery's? There are a few typo's in the features section btw (minmal, seriazlied).
- ohmygeek 12y ago
  We had evaluated Celery for our use case. There were a few things it couldn't do such as create queues dynamically with per queue rate limit. We chose Lua because the queue operations involve modifying more than one Redis data structure and this had to be atomic. We'll be adding more details on why we built this and how it compares with Celery.. :)
  - SEJeff 12y ago
    Can you please not do one thing Ask (celery author) constantly does? He breaks the celery API to a ridiculous degree between point releases. I've used celery for 5+ years and constantly have to hard fix the version I use in a requirements.txt or whatnot because the most seemingly minor point release results in broken tasks and a different api. Use semver or something sane to not make your users angry. Pretty please!
- armon 12y ago
  I'm also curious how this compares to Celery with rate limiting. One advantage of Celery is that you can use an AMQP broker which is probably better suited than Redis if you care about durability and at-least-once delivery.
  - ohmygeek 12y ago
    IIRC, Celery uses token bucket algorithm for rate limiting. SHARQ uses the leaky bucket algorithm (http://en.wikipedia.org/wiki/Leaky_bucket#The_Leaky_Bucket_Algorithm_as_a_Queue http://en.wikipedia.org/wiki/Leaky_bucket#The_Leaky_Bucket_A...). The advantage of leaky bucket is that, it enforces a constant flow of jobs out of SHARQ regardless of the spike in enqueue rate. Even though SHARQ uses Redis, it has been designed in a way to support at-least-once delivery. If you see the getting started section (http://sharq.io/docs/gettingstarted.html http://sharq.io/docs/gettingstarted.html), there is a Finish API which would acknowledge every job dequeued. SHARQ ensures that a job will be requeued back into the queue if it does not receive a Finish request.
makmanalp 12y ago
http://sharq.io/docs/faqs.html#when-should-i-use-sharq http://sharq.io/docs/faqs.html#when-should-i-use-sharq When people ask this question, the answer they're looking for is usually not "use Sharq when you need <definition of sharq>". They're looking for concrete examples or situations where one would require that.
inconshreveable 12y ago
It would be nice to credit me since I developed this approach (down to using Redis with Lua scripts) two years ago at Twilio: http://vimeo.com/52569901 http://vimeo.com/52569901 If you developed the approach entirely without having seen my talk then feel free to disregard this comment, and please accept my apologies for being presumptuous.

wc- 12y ago

Looks pretty cool, looking forward to playing around with it. How does this compare to celery w/ rate limiting of individual tasks? Celery can be pretty daunting (heavyweight perhaps?) sometimes, having a simpler alternative could be useful. However, I wonder if by the time you need rate limiting of queues, you will probably need additional "heavy" features as well. I'd love to see a deeper comparison of sharq's usage of redis vs celery's. For example, are there any performance gains by using sharq's lua commands or schemas vs celery's? There are a few typo's in the features section btw (minmal, seriazlied).

ohmygeek 12y ago
We had evaluated Celery for our use case. There were a few things it couldn't do such as create queues dynamically with per queue rate limit. We chose Lua because the queue operations involve modifying more than one Redis data structure and this had to be atomic. We'll be adding more details on why we built this and how it compares with Celery.. :)
- SEJeff 12y ago
  Can you please not do one thing Ask (celery author) constantly does? He breaks the celery API to a ridiculous degree between point releases. I've used celery for 5+ years and constantly have to hard fix the version I use in a requirements.txt or whatnot because the most seemingly minor point release results in broken tasks and a different api. Use semver or something sane to not make your users angry. Pretty please!
armon 12y ago
I'm also curious how this compares to Celery with rate limiting. One advantage of Celery is that you can use an AMQP broker which is probably better suited than Redis if you care about durability and at-least-once delivery.
- ohmygeek 12y ago
  IIRC, Celery uses token bucket algorithm for rate limiting. SHARQ uses the leaky bucket algorithm (http://en.wikipedia.org/wiki/Leaky_bucket#The_Leaky_Bucket_Algorithm_as_a_Queue http://en.wikipedia.org/wiki/Leaky_bucket#The_Leaky_Bucket_A...). The advantage of leaky bucket is that, it enforces a constant flow of jobs out of SHARQ regardless of the spike in enqueue rate. Even though SHARQ uses Redis, it has been designed in a way to support at-least-once delivery. If you see the getting started section (http://sharq.io/docs/gettingstarted.html http://sharq.io/docs/gettingstarted.html), there is a Finish API which would acknowledge every job dequeued. SHARQ ensures that a job will be requeued back into the queue if it does not receive a Finish request.

makmanalp 12y ago

http://sharq.io/docs/faqs.html#when-should-i-use-sharq http://sharq.io/docs/faqs.html#when-should-i-use-sharq When people ask this question, the answer they're looking for is usually not "use Sharq when you need <definition of sharq>". They're looking for concrete examples or situations where one would require that.

inconshreveable 12y ago

It would be nice to credit me since I developed this approach (down to using Redis with Lua scripts) two years ago at Twilio: http://vimeo.com/52569901 http://vimeo.com/52569901 If you developed the approach entirely without having seen my talk then feel free to disregard this comment, and please accept my apologies for being presumptuous.