7 ms·

Celery - Distributed Task Queue

70 points by albertzeyer 13y ago

jensenbox 13y ago
I have yet to read anywhere why using Postgres or Postgres with Django is a bad idea. Everything says a vague "suboptimal" without any real backing of that statement. Anyone know the real reasons behind that statement?
- bayesianhorse 13y ago
  If you are asking for postgres as a celeryd backend, it's not really what this dbms is meant to do... Redis is easy to install, run and manage (as long as you don't lean out of the window too much), so for celeryd it's preferable...
  - mattdeboard 13y ago
    Right, but "it's not what this was meant to do" is the kind of vague non-answer that the parent was talking about.
- gtaylor 13y ago
  The workers have to poll Postgres repeatedly at intervals, whereas a worker using a message broker (like RabbitMQ) connects and just waits for something to get tossed its way. Having your task broker separate from your database can save a lot of DB IO if you've got a larger number of workers or a busy task queue. RabbitMQ (and the other more full-featured message brokers) can also do a lot of really slick things with routing, prioritization, and all sorts of other goodies. Of course, this doesn't matter if you're running a small site with low traffic. I wouldn't get too caught up in worrying about broker selection unless you're cranking out a lot of jobs, or have special needs.
  - jensenbox 13y ago
    So if I am reading this right, what you are saying is that RabbitMQ is more 'push' and a DB is more 'pull' or rather 'poll'. Perhaps they should break the brokers down into those two groups. I seem to recall Redis having a pubsub style interface, which makes it much more push but still is a bit cloudy in my mind. Thanks for the reply!
memset 13y ago
Here is a question: I often find myself wanting to do background tasks within my application (say, a Flask app) without running celeryd. While celery is quite useful, I just want to spin up an application and have it Just Work without external dependencies. With celery, now I have to run celeryd, run some sort of queue (redis, rabbitmq, etc), and just worry about all of these pieces simply to execute long-running background tasks. Are there options for this? Akin to an sqlite for tasks, in python?
- JimmyL 13y ago
  I've read about https://github.com/tthieman/dagobah https://github.com/tthieman/dagobah recently, but I haven't used it...also, if you think celery is too heavy, you'll probably think this is too. I've also heard good things about https://github.com/binarydud/pyres https://github.com/binarydud/pyres. Despite all that, I usually just find myself running celery. I'll usually have a redis instance going anyways that I can use as a sub-optimal queue, and if it's a small enough site, I'll run a single worker instance on the same machine as the host.
- scott_w 13y ago
  Celery can use a database backend, such as Postgres. Maybe it can work with sqlite?
  - shuzchen 13y ago
    You definitely can use celery with sqlite via the sqlalchemy backend, but since rdbms' aren't very good task queues there are drawbacks: see http://docs.celeryproject.org/en/latest/getting-started/brokers/sqlalchemy.html http://docs.celeryproject.org/en/latest/getting-started/brok...
    - scott_w 13y ago
      That's true. However, as an answer to the above it may well be acceptable without needing a different library.
- whitehat2k9 13y ago
omarish 13y ago
We use celery pretty extensively. It's nice because you get the flexibility of multiprocessing, but also the ability to distribute tasks across many machines. There is definitely an overhead to get it going. Using redis as your messaging backend reduces this cost somewhat.. rabbitmq is overkill for most of the cases we've used celery. Have a look at rq if you're looking for something more lightweight: https://github.com/nvie/rq https://github.com/nvie/rq
pepijndevos 13y ago
What is there to see? Celery is great, but has been around for ages. Anything new I'm missing?
nemothekid 13y ago
I've been looking to replace our in-house postgres based task queue with something more suited for the task. What has been turning me off about celery/rq/qless is that they seem tied to the host languages (python/ruby), our backend is Go, and the Postgres mostly acts as a datastore. I'd love a somewhat dropin replacement where instead of reading/writing to pg, I'm reading/writing to redis/some other in memory store. What I want is the ability add a task with a priority and metadata, and be able to pop those tasks in O(1) time.
buster 13y ago
I have been using Celery in a recent python project with rabbitmq and it as a very good experience, celery works really nice and reliable. Would use it again!
ivankirigin 13y ago
I think RQ is simpler and better. http://python-rq.org/ http://python-rq.org/ Celery has a lot more control and features that you probably don't need.
- gtaylor 13y ago
  No knock to any contributors on here, but I'm not seeing why RQ is better. Documentation looks incomplete, seems to depend on Redis, the boilerplate doesn't appear to be any less complicated. Is there something I'm missing that this does much better?
- the_cat_kittles 13y ago
  I have tried to use RQ a couple times as a result of this thinking. Each time I end up going back to Celery because there always seems to be something I need that RQ doesn't have. For instance, in the most recent project, I wanted to execute tasks X minutes in the future- celery's eta kwarg is perfect for that, and I'm not sure how you would do it with RQ.
McGlockenshire 13y ago
Outside of using AQMP, what does this get me over using Gearman?
- gtaylor 13y ago
  If you're using Python and/or Django already and don't need to mix with other languages, Celery is pretty easy to work with, has excellent documentation, and is well-maintained. With that said, while there is no boundary to playing nice with other languages, it seems like only recently have people started to get serious about trying to do so with Celery (and sharing/supporting their work publicly). I haven't used Gearman personally, but if I had a polyglot project, I'd give it a serious look.
guiomie 13y ago
How is Celery different from RabbitMQ ?
- alexmic 13y ago
  Celery is a task framework. RabbitMQ is a queue server. Celery can use RabbitMQ as a message broker i.e a way to transfer the messages from your application to the workers.
rdtsc 13y ago
A simple solution (and I see a couple of requests here) is to use eventlet as the underlying concurrency library and then just spawn a green thread (or a green pool! to limit concurrency) to handle long running/background tasks.

70 points by albertzeyer 13y ago

I have yet to read anywhere why using Postgres or Postgres with Django is a bad idea. Everything says a vague "suboptimal" without any real backing of that statement. Anyone know the real reasons behind that statement?

bayesianhorse 13y ago
If you are asking for postgres as a celeryd backend, it's not really what this dbms is meant to do... Redis is easy to install, run and manage (as long as you don't lean out of the window too much), so for celeryd it's preferable...
- mattdeboard 13y ago
  Right, but "it's not what this was meant to do" is the kind of vague non-answer that the parent was talking about.
gtaylor 13y ago
The workers have to poll Postgres repeatedly at intervals, whereas a worker using a message broker (like RabbitMQ) connects and just waits for something to get tossed its way. Having your task broker separate from your database can save a lot of DB IO if you've got a larger number of workers or a busy task queue. RabbitMQ (and the other more full-featured message brokers) can also do a lot of really slick things with routing, prioritization, and all sorts of other goodies. Of course, this doesn't matter if you're running a small site with low traffic. I wouldn't get too caught up in worrying about broker selection unless you're cranking out a lot of jobs, or have special needs.
- jensenbox 13y ago
  So if I am reading this right, what you are saying is that RabbitMQ is more 'push' and a DB is more 'pull' or rather 'poll'. Perhaps they should break the brokers down into those two groups. I seem to recall Redis having a pubsub style interface, which makes it much more push but still is a bit cloudy in my mind. Thanks for the reply!

memset 13y ago

Here is a question: I often find myself wanting to do background tasks within my application (say, a Flask app) without running celeryd. While celery is quite useful, I just want to spin up an application and have it Just Work without external dependencies. With celery, now I have to run celeryd, run some sort of queue (redis, rabbitmq, etc), and just worry about all of these pieces simply to execute long-running background tasks. Are there options for this? Akin to an sqlite for tasks, in python?

JimmyL 13y ago
I've read about https://github.com/tthieman/dagobah https://github.com/tthieman/dagobah recently, but I haven't used it...also, if you think celery is too heavy, you'll probably think this is too. I've also heard good things about https://github.com/binarydud/pyres https://github.com/binarydud/pyres. Despite all that, I usually just find myself running celery. I'll usually have a redis instance going anyways that I can use as a sub-optimal queue, and if it's a small enough site, I'll run a single worker instance on the same machine as the host.
scott_w 13y ago
Celery can use a database backend, such as Postgres. Maybe it can work with sqlite?
- shuzchen 13y ago
  You definitely can use celery with sqlite via the sqlalchemy backend, but since rdbms' aren't very good task queues there are drawbacks: see http://docs.celeryproject.org/en/latest/getting-started/brokers/sqlalchemy.html http://docs.celeryproject.org/en/latest/getting-started/brok...
  - scott_w 13y ago
    That's true. However, as an answer to the above it may well be acceptable without needing a different library.
whitehat2k9 13y ago

omarish 13y ago

We use celery pretty extensively. It's nice because you get the flexibility of multiprocessing, but also the ability to distribute tasks across many machines. There is definitely an overhead to get it going. Using redis as your messaging backend reduces this cost somewhat.. rabbitmq is overkill for most of the cases we've used celery. Have a look at rq if you're looking for something more lightweight: https://github.com/nvie/rq https://github.com/nvie/rq

pepijndevos 13y ago

What is there to see? Celery is great, but has been around for ages. Anything new I'm missing?

nemothekid 13y ago

I've been looking to replace our in-house postgres based task queue with something more suited for the task. What has been turning me off about celery/rq/qless is that they seem tied to the host languages (python/ruby), our backend is Go, and the Postgres mostly acts as a datastore. I'd love a somewhat dropin replacement where instead of reading/writing to pg, I'm reading/writing to redis/some other in memory store. What I want is the ability add a task with a priority and metadata, and be able to pop those tasks in O(1) time.

buster 13y ago

I have been using Celery in a recent python project with rabbitmq and it as a very good experience, celery works really nice and reliable. Would use it again!

ivankirigin 13y ago

I think RQ is simpler and better. http://python-rq.org/ http://python-rq.org/ Celery has a lot more control and features that you probably don't need.

gtaylor 13y ago
No knock to any contributors on here, but I'm not seeing why RQ is better. Documentation looks incomplete, seems to depend on Redis, the boilerplate doesn't appear to be any less complicated. Is there something I'm missing that this does much better?
the_cat_kittles 13y ago
I have tried to use RQ a couple times as a result of this thinking. Each time I end up going back to Celery because there always seems to be something I need that RQ doesn't have. For instance, in the most recent project, I wanted to execute tasks X minutes in the future- celery's eta kwarg is perfect for that, and I'm not sure how you would do it with RQ.

McGlockenshire 13y ago

Outside of using AQMP, what does this get me over using Gearman?

gtaylor 13y ago
If you're using Python and/or Django already and don't need to mix with other languages, Celery is pretty easy to work with, has excellent documentation, and is well-maintained. With that said, while there is no boundary to playing nice with other languages, it seems like only recently have people started to get serious about trying to do so with Celery (and sharing/supporting their work publicly). I haven't used Gearman personally, but if I had a polyglot project, I'd give it a serious look.

guiomie 13y ago

How is Celery different from RabbitMQ ?

alexmic 13y ago
Celery is a task framework. RabbitMQ is a queue server. Celery can use RabbitMQ as a message broker i.e a way to transfer the messages from your application to the workers.

rdtsc 13y ago

A simple solution (and I see a couple of requests here) is to use eventlet as the underlying concurrency library and then just spawn a green thread (or a green pool! to limit concurrency) to handle long running/background tasks.