Nope. It is very specifically tailored to crawling. If you just need something distributed why not check out RQ [0], Gearman [1] or Celery [2]? RQ and Celery ar

Nope. It is very specifically tailored to crawling. If you just need something distributed why not check out RQ [0], Gearman [1] or Celery [2]? RQ and Celery are python specific. [0] : http://python-rq.org/docs/ http://python-rq.org/docs/ [1] : http://gearman.org/ http://gearman.org/ [2] : http://docs.celeryproject.org http://docs.celeryproject.org