We use celery pretty extensively. It's nice because you get the flexibility of multiprocessing, but also the ability to distribute tasks across many machines.

We use celery pretty extensively. It's nice because you get the flexibility of multiprocessing, but also the ability to distribute tasks across many machines. There is definitely an overhead to get it going. Using redis as your messaging backend reduces this cost somewhat.. rabbitmq is overkill for most of the cases we've used celery. Have a look at rq if you're looking for something more lightweight: https://github.com/nvie/rq https://github.com/nvie/rq