5 ms·

Large Problems in Django, Mostly Solved: Delayed Execution

58 points by mnemonik 16y ago

jnoller 16y ago
I can personally attest to the fact Celery is pretty awesome, and a nice compliment to your Django app. The author - Ask is also doing some awesome things with multiprocessing in general - check out http://pypi.python.org/pypi/billiard/ http://pypi.python.org/pypi/billiard/
pxlpshr 16y ago
We've been using Celery with Django for a solid 8 months or so. It's been a blessing for our needs, and extremely easy to scale.
tomazmuraus 16y ago
I use Celery myself as well and it's awesome. It supports multiple messaging queues and backends for storing results (RabbitMQ, MongoDB, AMQP, Redis, ... - I personally use Redis, because I don't need all the features and overhead from RabbitMQ) and also has a great documentation and friendly support on IRC. And if you watch the Celery project on github you can see that the author is basically working and improving the project on daily basis.
daredevildave 16y ago
How does celery differ in functionality than a message queue like RabbitMQ?
- kingkilr 16y ago
  Celery leverages a message queue (including Rabbit, which is how we deploy it), Celery provides an easy API to define tasks, push them onto your queue, and run them on workers.
jonasvp 16y ago
Celery together with RabbitMQ (or another message broker) is great for big/huge projects. For basic job queueing and delayed job execution, beanstalkd (http://kr.github.com/beanstalkd/ http://kr.github.com/beanstalkd/) is in most cases a better choice. It takes almost no memory, no configuration, and just works. I wrote a small library for Django that makes writing jobs for beanstalkd easier (http://github.com/jonasvp/django-beanstalkd http://github.com/jonasvp/django-beanstalkd). There's also a management command for starting (threaded) workers. Makes writing and calling jobs from Django very simple.
- asksol 16y ago
  Could you please elaborate why it is a better choice? I don't see how django-beanstalkd could take much less memory than celeryd. If it's about the broker (beanstalk), you could have implemented a beanstalk backend for celery as well, then you would have had a tested implementation of the worker part. Btw, If you would be interested in writing a beanstalkd backend for celery I would be happy to help you out!
  - jonasvp 16y ago
    Good point - I was under the impression that celery _only_ works with an AMQP message broker. Guess I should have read the description more closely before starting on django-beanstalkd. I'm happy with my lightweight model for now but I'll definitely look into celery again, when the need arises.
barnaby 16y ago
Do scheduling tools in a web framework protect you from having to write threading code yourself? Or is it for a completely separate kind of side-process than when I would want to write threads in my web app code? Also, is that like Quarts in the Spring Framework?
- steveklabnik 16y ago
  > Or is it for a completely separate kind of side-process than when I would want to write threads in my web app code? This isn't about threading, really. I'll explain it in terms of Rails's delayed_job because I'm not familiar with the details of Celery: * Let's say I want to send an email, via the function Mailer.deliver_notice(email) * With delayed_job, I turn this call into Mailer.delay.deliver_notice(email). This puts an entry into a Jobs table in the database that makes note of what I want to run, with all the details. * In the background, I have a daemon running that checks the Jobs table every five seconds for jobs that need to be run. It uses the information in the table to execute the deliver_notice function. * That's it. In many cases, the job will update some sort of field in your model that you can query to see if it's been run yet or not.
ebtalley 16y ago
It's a little off topic but I would like to take a moment and profess how much I LOVE the fact that hacker news has so many articles on all languages, popular or not.
jsdalton 16y ago
What are the advantages/disadvantages of using a messaging framework like Celery vs. handling asynchronous tasks in threads?
- bfirsh 16y ago
  Celery workers can run on any networked computer - it can be scaled infinitely!
nphase 16y ago
I'm a big fan of doing this with Gearman (http://gearman.org/ http://gearman.org/). Super simple interface, very easy to shove stuff into the background.

58 points by mnemonik 16y ago

I can personally attest to the fact Celery is pretty awesome, and a nice compliment to your Django app. The author - Ask is also doing some awesome things with multiprocessing in general - check out http://pypi.python.org/pypi/billiard/ http://pypi.python.org/pypi/billiard/

pxlpshr 16y ago

We've been using Celery with Django for a solid 8 months or so. It's been a blessing for our needs, and extremely easy to scale.

tomazmuraus 16y ago

I use Celery myself as well and it's awesome. It supports multiple messaging queues and backends for storing results (RabbitMQ, MongoDB, AMQP, Redis, ... - I personally use Redis, because I don't need all the features and overhead from RabbitMQ) and also has a great documentation and friendly support on IRC. And if you watch the Celery project on github you can see that the author is basically working and improving the project on daily basis.

daredevildave 16y ago

How does celery differ in functionality than a message queue like RabbitMQ?

kingkilr 16y ago
Celery leverages a message queue (including Rabbit, which is how we deploy it), Celery provides an easy API to define tasks, push them onto your queue, and run them on workers.

jonasvp 16y ago

Celery together with RabbitMQ (or another message broker) is great for big/huge projects. For basic job queueing and delayed job execution, beanstalkd (http://kr.github.com/beanstalkd/ http://kr.github.com/beanstalkd/) is in most cases a better choice. It takes almost no memory, no configuration, and just works. I wrote a small library for Django that makes writing jobs for beanstalkd easier (http://github.com/jonasvp/django-beanstalkd http://github.com/jonasvp/django-beanstalkd). There's also a management command for starting (threaded) workers. Makes writing and calling jobs from Django very simple.

asksol 16y ago
Could you please elaborate why it is a better choice? I don't see how django-beanstalkd could take much less memory than celeryd. If it's about the broker (beanstalk), you could have implemented a beanstalk backend for celery as well, then you would have had a tested implementation of the worker part. Btw, If you would be interested in writing a beanstalkd backend for celery I would be happy to help you out!
- jonasvp 16y ago
  Good point - I was under the impression that celery _only_ works with an AMQP message broker. Guess I should have read the description more closely before starting on django-beanstalkd. I'm happy with my lightweight model for now but I'll definitely look into celery again, when the need arises.

barnaby 16y ago

Do scheduling tools in a web framework protect you from having to write threading code yourself? Or is it for a completely separate kind of side-process than when I would want to write threads in my web app code? Also, is that like Quarts in the Spring Framework?

steveklabnik 16y ago
> Or is it for a completely separate kind of side-process than when I would want to write threads in my web app code? This isn't about threading, really. I'll explain it in terms of Rails's delayed_job because I'm not familiar with the details of Celery: * Let's say I want to send an email, via the function Mailer.deliver_notice(email) * With delayed_job, I turn this call into Mailer.delay.deliver_notice(email). This puts an entry into a Jobs table in the database that makes note of what I want to run, with all the details. * In the background, I have a daemon running that checks the Jobs table every five seconds for jobs that need to be run. It uses the information in the table to execute the deliver_notice function. * That's it. In many cases, the job will update some sort of field in your model that you can query to see if it's been run yet or not.

ebtalley 16y ago

It's a little off topic but I would like to take a moment and profess how much I LOVE the fact that hacker news has so many articles on all languages, popular or not.

jsdalton 16y ago

What are the advantages/disadvantages of using a messaging framework like Celery vs. handling asynchronous tasks in threads?

bfirsh 16y ago
Celery workers can run on any networked computer - it can be scaled infinitely!

nphase 16y ago

I'm a big fan of doing this with Gearman (http://gearman.org/ http://gearman.org/). Super simple interface, very easy to shove stuff into the background.