6 ms·
Celery 2.3 released
- Atilla 15y agoOne of the best releases so far!
- jws 15y agoWhat is Celery? Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready).
- StavrosK 15y agoAsk described it to me as "a library for discovering bugs in the multiprocessing library".
- asksol 15y agoYes, but note that multiprocessing rocks. It's just that celery uses it so extensively :) (and now also eventlet/gevent if that suits better)
- reinhardt 15y agoDoes Celery support task dependencies? Would be great to specify a DAG of tasks and have them run in parallel. Basically something like Oozie but not restricted to Hadoop.
- dangoldin 15y agoIf I understand your question correctly, you should be able to do this using callbacks.
- dtran 15y agoGreat work as always asksol!
- bryanh 15y agoMaybe it is just me, but I can never bring myself to setting up Celery and RabbitMQ and all for some Django apps that could sorely use it. There seems to be so many moving parts, last I checked there were like 3-4 different services to tie everything together... Anyone know of a simpler system or technique for achieving the same thing? Or perhaps, a better way to go about using Celery.
- bretthoerner 15y agoThe only moving parts are the broker (RabbitMQ, Redis, etc), your existing app (already moving), and some workers --- which for a simple case like yours is "python manage.py celeryd". It's worth it, in my opinion. Celery is one of the best-of-breed (even outside of Python) tools like SQLAlchemy.
- dtran 15y agoIMHO, setting up RabbitMQ and Celery, especially with django-celery, ends up being much simpler and less painful than rolling any kind of custom queue/async processing. In terms of moving parts, for most setups, you can put Celery and RabbitMQ under supervisord and almost never have to worry about them. If you don't need all the features of RabbitMQ or just don't want to deal with it, you can use any of the other broker backends: Redis (if you don't need persistence) or even with your default DB through the Django ORM.
- Pewpewarrows 15y agoFyi, redis has an appendonly option to provide persistence: http://redis.io/topics/persistence http://redis.io/topics/persistence
- asksol 15y agoYes, but I think what he was referring to was message acknowledgements, which Redis doesn't have. If a worker reserves a message, and is abruptly killed, then the message is lost.
- funksta 15y ago
- guildchatter 15y agoAnybody here use Celery on their production system? Is it reliable/stable? Any weird problems?
- chintan 15y agoAwesome tool! We are running some massively distributed tasks with it.
- StavrosK 15y agoWe use it for historious, it has never had any problems. I would definitely, definitely recommend it.
- matclayton 15y agoWe run it for mixcloud.com its awesome, saved us hundreds of hours of dev time easily.
- RyOnLife 15y agoWe use it/love it at Uber. Had a problem has been with the results building up in RabbitMQ and causing the queue to exceed it's memory allocation. We disabled results, and fortunately that's now the default with this release.
- enjo 15y agoWe use it in production and have for a couple of years. It's never once caused an issue.
- vae77 15y agoGreat news! Awesome software!
- thedangler 15y agoI have no idea when I would need to use it. Can someone give examples of when I might want to think of using it?
- baddox 15y agoYou can use it for any task which takes too long to fit into a web request. A simplistic example: You have a web form where a new user can choose a bunch of his/her interests or hobbies from a big list. The user clicks submit and then sees a list of other users on your site with similar interests. For small numbers of users and a simplistic algorithm, this could probably be done right in the web request (i.e. in your Django view), but with millions of users and a complex recommendation algorithm, you could have Celery do the task in a thread separate from the web request. You could then immediately forward the user to a page that says "Your recommendations are being calculated." The user can come back later and see if his/her recommendations are done yet. I assume you could even do some AJAX checks that would load in the recommendations once the Celery task is done, or even display a progress bar.
- enjo 15y agoDjango-celery even comes with a built-in view to do that polling;)
- enjo 15y agoWe use it extensively, essentially for anything that doesn't need to happen "just this second". It's also replaced everything we used to do in cron, as it easy to carry around in source control. Most of our emails, calculating expensive things (we have some match-making algorithms that run as tasks), sending notifications and all sorts of fun things are all done in tasks. By moving as much of our code into celery as possible it's been much easier to scale (adding a new node is trivial) and our pages are much more responsive. That Celery makes it all so very easy to do is icing on the cake.
- mahmud 15y agoSo, like Quartz. http://quartz-scheduler.org/overview/index.html http://quartz-scheduler.org/overview/index.html