I've had good success using Celery to parallelize tasks/jobs in python. www.celeryproject.org Also, it has a very nice concept called canvas that allows you t

I've had good success using Celery to parallelize tasks/jobs in python. www.celeryproject.org Also, it has a very nice concept called canvas that allows you to chain/combine the data/results of different tasks together. It also allows you to switch out different implementations of the communication infrastructure that Celery uses to communicate and dish-out tasks.